Story: Synthetic data scope and binding: system/tenant/party levels, bound vs sandboxed
Table of Contents
This page documents a story in Sprint 25. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
Goal
Give synthetic market data generation two orthogonal primitives – scope (system/tenant/party) and binding mode (bound/sandboxed) – so a tenant can choose to share one synthetic reality across its parties (ACME's case), and so synthetic data can later be generated for testing (new ORE configs, quant models) without silently overriding real system behaviour.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent sprint | Sprint 25 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-08-09 |
Acceptance
- market_data_generation_config carries a scope (system/tenant/party) and a binding_mode (bound/sandboxed), each correctly nullable/FK'd per scope level
- Activating a bound config that narrows an already-covered instrument at a wider scope is rejected, requiring explicit supersession – never silent override
- Sandboxed configs generate and publish data end-to-end under a distinct namespace, provably excluded from bound-feed resolution used by real consumers
- ACME's 3 existing party-level configs (UK/US/HK) are consolidated into a single tenant-level bound config; ACME parties share one synthetic reality
- Market Simulator tree filters by viewer scope (own party's configs plus any tenant/system configs visible to them) instead of showing every party's configs unfiltered
- Design doc records the future workspace-scoped binding direction (live workspace = real feeds, user workspaces bind to sandboxed synthetic runs) without implementing it
Plan
Problem
market_data_generation_config is party-scoped only, with no distinction
between "authoritative shared reality" and "sandboxed test data". This
causes ACME's three subsidiary parties (UK/US/HK) to each run their own
diverging synthetic setup instead of sharing one reality, and the Market
Simulator UI leaks every party's configs to any viewer regardless of who
is asking.
Two orthogonal axes
- Scope –
system/tenant/party. Defines the sharing radius: who consumes the same generated data. Corresponding tenant_id/party_id columns are nullable per level (system: both null, tenant: tenant_id only, party: both set). - Binding mode –
bound/sandboxed. Defines authority:boundconfigs are resolved into the feed real consumers see;sandboxedconfigs generate and publish data reachable only by explicit selection, never touching bound-feed resolution.
These are independent: a config can be any (scope, binding_mode) pair.
Conflict resolution
At most one bound config may be active per (instrument, scope) at a
time. Activating a narrower-scope bound config for an instrument already
covered by a wider-scope bound config is rejected at activation time –
the caller must explicitly supersede (stop/retire) the wider one first.
No silent narrowest-scope-wins resolution: ambiguity must never reach a
consumer. sandboxed configs never participate in this check – they
cannot conflict with anything.
Sandbox primitive (partial, by design)
sandboxed configs are runnable end-to-end: they generate and publish
under a distinct namespace, provably excluded from the bound-feed
resolution path. What is deliberately not built: any UI/UX for a
consumer (an ORE run, a model test) to select and attach to a specific
sandboxed config/run. That consumption side is real, separate work.
ACME migration
Consolidate the existing UK/US/HK party-level configs into a single
tenant-level bound config; retire the three party-level ones. ACME's
holding-group parties then share one synthetic reality.
Visibility fix (folded in, supersedes existing capture)
The Market Simulator tree currently shows every party's configs unfiltered to any viewer – this is a scope leak, not just noise. Once scope exists on the entity, filter the tree by viewer scope: the viewer's own party's configs, plus any tenant/system-level configs visible to them. This supersedes the open capture "Market Simulator: scope folders to current party, clarify feed-start UX" (compass show 275824BB-0DCE-4EA4-964D-9C86B0498509).
Future direction (documented only, not built)
ores.workspace today is user-scoped UI layout/preference state, not a
data-binding concept. It may later become a binding target in its own
right: a "live" workspace holding real (bound) feeds, with users able to
create their own workspaces and bind sandboxed synthetic runs into them
for undisturbed experimentation. The sandbox namespace introduced here
should stay generic (not FX-spot-specific, not tied to a single scope
level) so a later workspace-scoped binding can reuse it without rework.
No workspace changes are made in this story.
Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Add scope (system/tenant/party) and binding_mode (bound/sandboxed) to market_data_generation_config | DONE | 2026-08-04 | 2026-08-04 | Extend the market_data_generation_config entity model with a scope enum and nullable tenant_id/party_id per level, plus a binding_mode enum (bound/sandboxed). Regenerate across all layers (domain, table I/O, generator, repository, Qt). Add activation-time validation rejecting a bound config that narrows an already-covered instrument at a wider scope. |
| Run sandboxed synthetic configs end-to-end under a distinct namespace | DONE | 2026-08-04 | 2026-08-05 | Wire ores.synthetic.service to generate and publish data for sandboxed configs under a namespace distinct from bound feeds, and prove the bound-feed resolution path used by real consumers never picks up sandboxed output. No consumer-side selection UI in this task. |
| Consolidate ACME's UK/US/HK party configs into one tenant-level bound config | DONE | 2026-08-05 | 2026-08-05 | Pick/merge one canonical config from ACME's three existing party-level synthetic configs, promote it to tenant scope with binding_mode=bound, and retire the other two party-level configs. Verify all three ACME parties then observe the same synthetic reality. |
| Filter Market Simulator tree by viewer scope (supersedes existing capture) | DONE | 2026-08-07 | Market Simulator currently shows every party's synthetic configs unfiltered to any viewer. Filter the tree to the viewer's own party's configs plus any tenant/system-level configs visible to them, using the new scope field. Supersedes capture 275824BB-0DCE-4EA4-964D-9C86B0498509 (Market Simulator: scope folders to current party, clarify feed-start UX) – close/link that capture as superseded once this lands. | |
| Add synthetic market simulator commands to ores.shell | DONE | 2026-08-05 | 2026-08-08 | Add shell commands to list folders, configs, and start/stop feeds for synthetic market data (PR #1914) |
| Add a stable code identifier to market simulator folders | DONE | 2026-08-08 | 2026-08-09 | Task for: Synthetic data scope and binding: system/tenant/party levels, bound vs sandboxed |
Decisions
- Conflict resolution is reject-at-activation, not resolve-at-read (narrowest-scope-wins was considered and rejected: two feeds silently generating for the same instrument risks confusing debugging and wastes compute).
- Sandbox mode ships runnable (generate + publish under a namespace), not merely a schema placeholder – validates the primitive actually works end-to-end, even with no consumer UI yet.
- ACME's three party configs are consolidated in this story, not deferred – otherwise the story ships the primitive but ACME's UX is unchanged.
scope=/=binding_modeare hand-written C++ enums (following the existingdomain::asset_classprecedent), not the codegenores.cpp.enumfacet – that facet exists but has no live usage yet in the codebase; introducing its first real usage was out of scope for this task.- The reject-at-activation conflict check lives in the database as a hand-written insert-trigger paste block (cross-row logic doesn't fit the per-column FK validation mechanism), not in service-layer C++ – keeps the guarantee true regardless of write path (Qt, shell, direct SQL), consistent with how version conflicts and tenant validation already work in this trigger.
- Qt enum columns need a
formatter(a small hand-written label function) wired via the Columns table – the generated Qt list-model code'sstringcolumn type assumes the underlying field actually is astd::string; there is no generated enum-to-QString path yet. - Sandboxed-feed exclusion from the bound-feed path is enforced two independent ways – a distinct NATS subject namespace and no feed_binding row – rather than relying on just one. Either mechanism alone would be sufficient today, but the redundancy means a bug in one doesn't silently collapse the guarantee.
- Live-testing the primitive (not just unit tests) surfaced a real gap
a unit test alone would have missed: the JetStream stream backing
synthetic ticks only covers the pre-existing subject pattern, so
publishing to a new subject with no matching stream crashes the
service. Worth remembering for any future new-subject work in this
service: a new producer subject needs its own
ensure_stream, not just a passing unit test. - The conflict-guard trigger from the first task was fixed to no longer block two enabled bound configs at the same party – an established, legitimate, pre-existing pattern (multiple named containers per party, e.g. "Basic"/"Realistic"), broken by the original over-broad guard and only surfaced by actually provisioning ACME rather than by unit/pgTAP tests alone.
- The guard's remaining granularity is per-tenant, not per-theme: promoting one container to tenant scope requires retiring every other enabled bound party-scope container in that tenant, not just duplicates of the same theme. Consolidating ACME meant retiring all six pre-existing party-level containers (two themes x three parties), not just the two directly-duplicated ones – a real, known consequence of the guard having no per-instrument visibility (see the trigger's own comment). Live-testing against real ACME data is what surfaced this; it would not have been obvious from the original design discussion alone.
- RLS read-widening (added for the container in the first task) had to
be extended one layer down to
fx_spot_generation_configandgmm_component– promoting the container's scope alone does not make its sub-configs visible to other parties; their ownparty_id-based RLS needed the identical read-widen/write-stays- strict treatment, joined through to the parent container's scope. - No new tooling/API was built to make consolidation itself repeatable – executed as documented, reproducible SQL against a live database, going through the same versioned-insert path (and hence the same conflict guard) the app's own writes use. Automating this into provisioning, or exposing it as a "supersede and consolidate" action, is future work.
- Found and fixed four pre-existing bugs while getting ACME
provisioning to actually run (two schema-drift issues in
ores.marketdataunrelated to this story, one regression in this story's own first task): each was small, contained, and directly blocking, so fixed inline rather than deferred as separate captures. - Shell command tokens resolve by exact match only (UUID, folder
name, ore key, source_name) – no substring/partial matching, which
was considered and rejected as a recipe for confusion; ambiguous
tokens list every candidate (folder path / source_name) so the user
can pick the full token. Multi-word names use the REPL quoting idiom
(
"synthetic list folders --name \"2026 Realistic\""), consistent with--tenant-name "Acme Corporation"elsewhere in the shell. - Folders are the one market-simulator entity without a stable
machine-readable identifier (no
code, unlike tenantcode/ configname/ feedore_key). A storedcodecolumn was considered and rejected: the codebase already derives machine tokens from display names (source_nameis the config-name slug plus the pair, built in publish_from_dq), and folder names are themselves derived from DQ dataset names at publish – not user-renamed – so a stored code would have no rename-stability consumer, would drift from the name, and would force schema regeneration across every layer. Follow-up task folder-code-identifier is re-scoped to derive the identifier at resolve time: each folder's codename is its slug (lower(replace(name, ' ', '')), mirroringsource_name), and folders resolve as uuid -> exact name -> codename path. Codename paths are standard, filesystem-style paths walked component-by- component from the tree root (2026realistic/fx,synthetic/2026realistic/fx), so same-named folders under different collections are addressed by path; a bare codename (fx) matches at any depth and reports ambiguous, listing each candidate's codename path. A ' > '-joined display-path token form ("2026 Realistic > FX") was considered during implementation and rejected as messy and untypable.
Out of scope
- Any UI/UX for a consumer (ORE run, model test) to select and attach to a specific sandboxed config/run.
- Workspace-scoped bindings –
ores.workspaceis not touched by this story; only documented as a future direction. - System-level (cross-tenant) configs being exercised end-to-end by any real scenario – the scope value is supported by the data model and resolution logic, but no current tenant needs it activated.