Story: Synthetic data librarian support: FX foundation
Table of Contents
This page documents a story in Sprint 22. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
Goal
When a party is created, the librarian gives it a usable synthetic data
configuration out of the box, so downstream generation, publishing, and
consumption work without manual setup per party. The configuration model
must span asset classes generically, but the only asset class wired up
end-to-end in this story is FX (spot). The db tables (e.g.
ores.synthetic.market_data_generation_config and its sub-configs) and
the entity's own publishing/NATS eventing are generated via
ores.codegen as usual. The DQ publish-from-dq wiring specifically
(artefact-type row, publish function, handler, registrar) is
hand-written for this story, following the existing report_definitions
instance as the template — generalising it into a reusable codegen
facet is deferred to
a dedicated backlog story.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent sprint | Sprint 22 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. Follow-on work tracked in the two spun-off backlog stories. |
| Last touched | 2026-07-03 |
Acceptance
[X]Creating a party (via librarian) results in a usable synthetic data configuration being available for it, covering at least FX spot. Verified: provisioning BARCLAYS PLC via the standardores.shellrecipe yields two enabled, ticking FX configs with zero manual steps.[X]The synthetic data configuration db schema and generated code (entity, repository, NATS publishing) are asset-class-generic — adding a new asset class means adding a sub-configuration, not reworking the container/infrastructure. Unchanged from the existingores.syntheticcodegen models; this story only added the DQ publish-from-dq wiring around them.[X]The synthetic FX config entity itself (domain, repository, NATS eventing) is produced viaores.codegen, as usual for any entity. The DQ publish-from-dq function/handler/registrar are hand-written for now (see Decisions) — that gap is tracked separately (Codegen facet: generate the DQ publish-from-dq pattern), not a blocker for this story.[X]FX synthetic data generation, once configured, publishes observations through the existingores.marketdataNATS eventing path. Confirmed via liveores.synthetic.servicelogs: both feeds auto-start and continuously publish onsynthetic.v1.tick.synthetic.{eurusd,gbpusd}.
Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Scaffold story: Synthetic data librarian support: FX foundation | DONE | 2026-07-02 | Break the story down into the full set of implementation tasks: db schema/codegen models for the synthetic data configuration container and FX sub-config, librarian defaulting logic on party creation, publishing/NATS eventing, and verification. Docs-only planning task. | |
| Design the publish_from_dq codegen facet | ABANDONED | 2026-07-02 | Split off to a dedicated backlog story (Codegen facet: generate the DQ publish-from-dq pattern) — too substantial to embed in this story. | |
| Implement the publish_from_dq codegen facet | ABANDONED | 2026-07-02 | Split off alongside the design task, same backlog story. | |
| Model synthetic FX spot config as a DQ artefact type | DONE | 2026-07-02 | 2026-07-02 | Hand-write the DQ artefact-type wiring for the synthetic FX config (artefact-type row, shadow table, publish function, handler, registrar), following report_definitions as the template — no codegen facet, that work is deferred to the backlog story above. |
| Seed FX spot synthetic config data and register the bundle | DONE | 2026-07-02 | 2026-07-02 | Seed the synthetic FX artefact table (EUR/USD, GBP/USD) and add the dataset as a member of the existing ore_analytics bundle (display_order 20, right after ore.report_definitions at 10) — corrected from an earlier, unnecessary separate bundle. |
| Decide the party-creation defaulting mechanism | DONE | 2026-07-02 | 2026-07-02 | Corrected: no new trigger or wizard phase needed — riding the ore_analytics bundle means every existing "publish reports" call site (CLI provisioning script, Qt wizard) already picks up the FX configs for free. The earlier separate-bundle + new-wizard-phase approach was reverted. |
| End-to-end verification: party to published FX ticks | DONE | 2026-07-03 | 2026-07-03 | Create a party, apply the ore_analytics bundle per the existing provisioning flow, and confirm a usable synthetic FX spot config is created and ores.synthetic.service picks it up and publishes ticks over the existing ores.marketdata NATS eventing path. |
| Wire synthetic FX bundle publish into Party Setup wizard (after Reports) | ABANDONED | 2026-07-02 | 2026-07-02 | Reverted: unnecessary. Adding the dataset as an ore_analytics bundle member (previous task) means the existing "publish reports" call sites already cover it — no new wizard phase or separate bundle needed. |
Decisions
- The DQ Librarian bundle-publish pipeline (DQ Bundle Publication Pipeline) is the mechanism this story builds on, not a new concept — synthetic FX config becomes one more publishable artefact type in that pipeline.
- Most of the pipeline's schema layer is already codegen-generated
(bundle, junction, artefact-type, shadow artefact table); the actual
codegen gap is the publish-from-dq SQL function + NATS handler +
registrar wiring, hand-written 25 times today. A generic codegen
facet to close that gap was judged too substantial to build inside
this story — it's deferred to
Codegen facet: generate the DQ publish-from-dq pattern in the
product backlog. This story hand-writes its publish-from-dq
artefacts (a 26th copy, using
report_definitionsas the template) so it isn't blocked on that larger effort. - There is no existing mechanism to trigger anything automatically on
party creation (
party_service::save_partyis a plain upsert); "the librarian gives it a usable config" in the Goal will be satisfied via the Librarian's existing manual bundle-Apply action unless the defaulting-decision task concludes an automatic trigger is required. - Defaulting mechanism decided, corrected twice: the synthetic FX
dataset (
synthetic.fx_spot_configs) is a member of the existingore_analyticsbundle, atdisplay_order 20right afterore.report_definitions(10) — not a separate bundle. An earlier attempt built a separatesynthetic_market_databundle plus a new automatic Phase 3.5 inPartyProvisioningWizard; both were reverted once it became clear a real test-provisioning run (viaores.shell) never touched either. - That test run surfaced a deeper problem: neither party-provisioning
call site (Qt
PartyExecutePage::startReportInstallorores.shell'sprovision_commands::process_party"reports" phase) actually publishes theore_analyticsbundle at all — both bypassdq.v1.bundles.publishentirely with a bespoke list-templates-then-create-per-record RPC path, predating (and never migrated onto) the publish-from-dq infrastructure thatreport_definitionsis otherwise fully wired into. Riding bundle membership alone therefore has no effect on either provisioning path as things stand. Filed as a separate story, Unify report definitions provisioning onto the standard publish-from-dq path, since fixing it properly (in both places, without regressing the CLI's per-report-name--reportsselection) is bigger than this story's scope. - Pragmatic interim fix for this story: added a one-off, hand-written
step to
ores.shell'sprovision_commands.cpp([4/5] Publishing synthetic FX spot configs) that callsdq.v1.bundles.publishwithbundle_code = "ore_analytics"andopted_in_datasets = ["synthetic.fx_spot_configs"]only — this is explicitly acknowledged as not data-driven (it's dataset-specific C++, the same class of problem as the bundle above), kept narrow and temporary until the unification story above replaces it with a single generic bundle-publish call covering allore_analyticsmembers. The Qt wizard path is not yet fixed at all (see Out of scope) — onlyores.shellprovisioning was exercised in this story's testing.
Out of scope
- Asset classes other than FX (spot) — infrastructure is generic, but only FX is wired end-to-end.
- New generation methodologies beyond what the sprint 21 approach analysis and FX spot PoC already established.
- Fixing the Qt
PartyProvisioningWizardpath to also publish the FX config — onlyores.shell's CLI provisioning was wired and tested in this story. Both call sites get fixed properly together in Unify report definitions provisioning onto the standard publish-from-dq path.