Task: Bind ores.synthetic entities to profiles; verify zero-diff regen
Table of Contents
- 1. Goal
- 2. Status
- 3. Acceptance
- 4. Plan
- 5. Notes
- 5.1. Binds (2026-09-07, phases 2-3 close)
- 5.2. Acceptance runs (2026-09-07, phase 4, in progress)
- 5.3. Investigation outcome and close (2026-09-08)
- 5.4. Live system test (2026-09-08, ores.shell)
- 5.5. Post-merge confirmation rerun and close (2026-09-08)
- 5.6. Baseline probe (2026-09-07, phase 1 start)
- 5.7. Survey classification (2026-09-07, phase 1 close)
- 6. Test Scenarios
- 7. PRs
- 8. Review
- 9. Result
This page documents a task in the Entity classification and drift baseline: ores.synthetic story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Survey every ores.synthetic entity against the profile catalogue, rebind matches, regenerate, and require zero additional diff against the pre-rebind checked-in output.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Entity classification and drift baseline: ores.synthetic |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-09-07 |
3. Acceptance
- Every entity discovered via the synthetic catalogue entry is surveyed against all Variability Profiles.
- Every matching entity is rebound and regenerated with zero additional diff versus its pre-rebind checked-in output.
- No-match entities are recorded with their actual feature values, not silently dropped.
- Any pre-existing drift the regen+diff surfaces is fixed, not worked around. See the related capture on fx_spot_generation_config's system-tenant fallback SQL codegen drift, and resolve it here if it falls within this catalogue entry's scope.
- Full local build and ctest green after regeneration.
4. Plan
Unit plan, written at start on 2026-09-07. The canonical method is the compass-codegen-fix-drift skill (steps 1-8); the byproduct-sweep and two-mode drift rules of the regen-byproduct-hygiene memory are in force throughout.
- Phase 1 — baseline and survey (skill steps 1-2). Probe done (see Notes); survey every model file, classify by metatype, compute resolved features per entity.
- Phase 2 — resolved features and binding (skill steps 3-4). Bind
matches with
:profile:, record every no-match with its actual feature values. - Phase 3 — per-entity regeneration (skill step 5). One entity at a time through the scripted pipeline: baseline regen, bind, regen, require the new diff to be an exact subset of the baseline. Regenerate a second time and require byte-identical output.
- Phase 4 — generated-code review, build, tests (skill step 6).
- Phase 5 — registrar wiring check, gate extension, PR (skill steps
7-8). The history-provider families materialized by the baseline
probe must be composed into the registrar fan-out (or recorded as
deliberately unwired); then the registry entry and the
--allrun.
5. Notes
5.1. Binds (2026-09-07, phases 2-3 close)
Eight binds, one commit per entity, each verified by the
asserted-replacement edits, the diff-subset check against the baseline
(zero additional diff), and an idempotent second regen:
yield_curve_process_type= simple-lookup (a79f8da03f): output-neutral bind; the Qtchanged_eventdeclaration follows the sibling lookups and is output-neutral. Controller lands thewireDetailDialogCommonera-sync.folder= self-referencing-hierarchy (8cf9600273, corrected by 35a7558c74): the bind materializes the canonical hierarchy machinery (recursiveores_synthetic_folders_hierarchy_fn, theget_hierarchychain, eventing test). Correction: the first bind dropped the explicit:has_parent_id:the template keys on, which deleted the machinery and broke the feed-control flow'sresolve_subtreeconsumer; the refdata counterparty org - the profile's reference entity - declareshas_parent_idexplicitly alongside the profile, and folder now does the same. The hand-written stack it replaces was already canonical-shaped (the consumer compiles unchanged), not superseded-era.yield_curve_process_parameter_definition= uuid-surrogate-lookup (e4d9237570): the profile'shas_change_reason_cacheflips the feature on; the controller gains theChangeReasonCacheplumbing, the survey's predicted phase-3 arbitration.fx_spot_generation_config,gmm_component,ir_curve_generation_config,ir_curve_template_entry,market_data_generation_config= uuid-identified-lookup (db42f8f766, 26c00d607d, 161bbb0794, 47fcf71e22, 87394fe434): clean binds; each eventing integration test materializes;market_data_generation_configkeepsnullable_tenant_id, nullableparty_id, and the enum columns explicit/out-of-drawer per the recorded notes.ir_curve_generation_config_process_parameter_value: NO-MATCH stands; era-sync signed off (b7add05b35, the doc-comment equals-sign restoration, the compute-unit regex fix class).
The history-provider registrar families landed as one commit (be36783d17) with all nine providers composed into the registrar fan-out's history.v1.get block on the refdata shape (skill step 7) and the cmake component lists regenerated — the drift gate cannot see an unwired function, so the wiring and the family land together.
Zero drift reached 2026-09-07: check_component_drift.py --component
synthetic exits 0 with a clean tree (no tracked diff, no untracked
materialization).
5.2. Acceptance runs (2026-09-07, phase 4, in progress)
Full build exit 0 through ores.qt.exe after two fixes at source: the core tests target lacked the eventing libs the generated tests include (CMakeLists, the refdata/compute/trading pattern), and the plugin's parameter-definition controller construction gained the change reason cache the uuid-surrogate-lookup bind turns on.
First ctest run (NATS down): 64/71, every failure = NATS connect refused - environmental. Second run (NATS up): 67/71.
ores.trading.core,ores.reporting.core,ores.compute.corenow pass.ores.synthetic.core.tests: 6 eventing tests fail - the generated seeding's INSERT violates theparty_isolationRLS onores_synthetic_market_data_generation_configs_tbl(and folders). Root cause: the mdc org's party_id generator snippet always stamps a fixed uuid, so the seeded ancestor row's party does not match the test session context the RLS checks. The text-keyyield_curve_process_typeeventing test passes. Fix belongs in the generated seeding / the org generator snippet, not the RLS.ores.synthetic.service.tests: 1 failure,folder_cascade_starts_and_stops_both_kinds_with_per_kind_counts, NATS request timed out - needs the full fleet up (7 units were still not ready: 5 compute wrappers, iam.service, refdata.service).ores.iam.core.tests/ores.refdata.core.tests: failed with no detail captured in the run excerpt; their publish tests ran very slowly (10-30s each) - investigate with the compass-code-investigate-test-failure loop.
Next: investigate the four suites (enable logging, per-test logs),
fix at source, rerun to 71/71, then skill step 8 (KNOWN_DRIFT_FREE
registry entry + --all) and the PR.
5.3. Investigation outcome and close (2026-09-08)
Three defects under the RLS failures, each fixed at source (fce56925b4):
- The seven hand-era party-isolation policies moved to the house read-side shape (visible_party_ids membership); each policy's container-widening logic is preserved.
- The ir_curve_generation_config check-constrained natural keys
(index_family, tenor, role) now declare
:no_generator_suffix:; the generator's uniqueness suffix was producing check-invalid values (sofr-1). - The ir_curve_template_entry instrument_code generator now emits the catalogue code (IRS), not a display name.
iam.core and refdata.core failures did not reproduce on a steady fleet (environmental); the folder-cascade NATS timeout was the not-yet-started fleet. Final evidence: full build exit 0, ctest 71/71 (908s), roundtrip clean, cmake component lists clean, check_component_drift –all exit 0 with synthetic in the registry on a clean tree. The RLS policy file also carries the coordinating environment's concurrent null-tenant widening, merged in place.
5.4. Live system test (2026-09-08, ores.shell)
The first pass exposed a real coverage gap: the shell had no per-entity CRUD commands for the synthetic entities, so the CRUD workflow could not be exercised. Nine command units were added (one per entity: get/add/delete/history over the generated protocol subjects, following the trading reference-entity pattern; an aggregator wires them), and the core registrar's three remaining hand-wired entity blocks (mdc, fx_spot, gmm) were swapped to their generated sub-registrars so the history subjects are actually subscribed. Five orgs also gained the Table display sections the JSON-to-org migration had dropped (their list converters rendered empty shells) - the trade_party_role defect class from the trading unit.
Full CRUD verified live against the recreated database:
- CREATE: yield_curve_process_type, market_data_generation_config, fx_spot_generation_config, ir_curve_generation_config, yield_curve_process_parameter_definition, gmm_component, ir_curve_template_entry (FK chain into the shell-created ir config), and process_parameter_value (FK chain into the ir config and the seeded parameter definitions) all insert; the insert triggers' existence checks and check constraints validated each row.
- READ: paginated get on every entity renders live tables.
- UPDATE/history: version history renders per entity.
- DELETE: by code (text key) and by uuid; the follow-up gets confirm removal.
- The feed control plane (start/stop cascade over the folder hierarchy) was exercised earlier in the same session.
- The gap-closing pass added: folder create via shell, the update path (add with the same natural key bumps the version - UPDTEST history shows versions 1 and 2), and per-entity history reads on all nine entities with their own keys.
- Finding recorded as a capture (party GUCs leaking across pooled connections): rows written by a session without a party context are stamped with the nil party uuid and are invisible to subsequent party-filtered reads through the service pool - the same session saw 26 party-scoped rows, then 0, with identical credentials, while the same read as the service role with explicit GUCs returned all rows. The RLS policies and the bind's data are correct; the defect is the per-connection GUC hygiene in the service layer, outside this task.
Ran on this environment's fleet against the recreated database, before merge (the post-merge confirmation rerun closes the story):
provision system/provision tenant --source acmecompleted end to end; the synthetic data steps (synthetic.themes.realistic_2026,synthetic_ore_samples_2016,system_market_data.bindings,synthetic_feeds) wrote through the regenerated repositories and the aligned RLS policies.synthetic list foldersrenders the 11-node hierarchy (the restoredores_synthetic_folders_hierarchy_fnchain reads live data).synthetic list configslists the two bound configs with scope, binding mode and party.synthetic start folder synthetic/2026realistic/fx/fxratesstarted the cascade (34 feeds running) through the feed-control flow'sresolve_subtreeover the regenerated repository.synthetic stop folderstopped exactly the 13 FX feeds, leaving the 21 IR curve feeds running;synthetic list feedsconfirms.synthetic validate-vintageresponds correctly (the tested FX feed has no vintage entries - informational, not a failure).synthetic generateis not usable post-provisioning: it tries to create a second root party per tenant and the unique index rejects it. Pre-existing flow precondition, unrelated to the bind; noted as a follow-up question for the generation command's intended sequencing.
5.5. Post-merge confirmation rerun and close (2026-09-08)
The post-merge rerun exercised the nine-entity CRUD sweep through
ores.shell against merged main (4118c5ce88), this environment's
fleet and the seeded database.
- CREATE/UPDATE: shell add on every entity wrote its row through the
generated repositories: the folder seed and its child under the
restored hierarchy; the process parameter definition under
PMERGEandPmergeConfig(market_data_generation_config); the ir/fx/gmm/ tpl/ppv FK chain (each insert passed the insert-trigger existence checks and the check constraints); thePMERGEprocess type re-add bumped the version to 2 – the update path still renders versions 1 and 2 on history. - READ: paginated get renders live tables with values on all nine entities; per-entity history reads render with their own keys (gmm fa22e023-9ac8, tpl 19b706a5-77cd, ppv 87efc4fc-a317 and siblings).
- DELETE: the full pass, leaf-first, through the shell delete verbs
(ten rows, including the
UPDTESTandShellTestConfigresidue rows from the pre-merge passes); follow-up gets confirm zero rows per entity. The acme synthetic scope is back to the seed state (the 23 seeded parameter definitions under the ddl role remain).
The rerun caught one genuine post-merge defect, the same class the
pre-merge pass fixed five times over: the ir_curve_template_entry
and ir_curve_generation_config_process_parameter_value list
converters still rendered empty shells. Their get/history footers
reported the correct totals, but no column headers and no rows
appeared. Root cause: the JSON-to-org migration dropped the ** Table
display section from these two orgs – the pre-merge fix covered
folder, fx_spot_generation_config, gmm_component,
ir_curve_generation_config and market_data_generation_config –
and with no section, codegen emits a zero-column fort converter. The
read path, service, RLS policies and data were healthy throughout: a
psql replication of the service queries under the service role with
the same GUCs returned the rows. Fixed at source: both orgs gained
their Table display sections (columns follow the Qt model labels for
the template entry; the config/parameter/value identity for the
parameter value row), the two models regenerated at
ores.cpp.domain (only the two converters change on disk), and the
rerun renders rows on both entities.
Closes the story per the 2026-09-07 done-shape decision.
5.6. Baseline probe (2026-09-07, phase 1 start)
Probe: check_component_drift.py --component synthetic (two-mode
rule, mode B) on the untouched tree; rc 1 as designed. Full diff
preserved at /tmp/drift_synthetic_baseline_20260907.log.
- 9 models regenerate at address
ores:folder,fx_spot_generation_config,gmm_component,ir_curve_generation_config,ir_curve_generation_config_process_parameter_value,ir_curve_template_entry,market_data_generation_config,yield_curve_process_parameter_definition,yield_curve_process_type. - 22 tracked files drifted, all template-era sync, no hand-edits
suspected: the three Qt controllers (
YieldCurveProcessType,IrCurveTemplateEntry,GmmComponent) and their headers predate thewireDetailDialogCommontemplate refactor (~59 lines each); three api domain headers carry small include/doc deltas; fiveores.sql/create/syntheticfiles carry 2-line era deltas; five repository headers lose one blank line; themarket_data_generation_configmapper gainsrfl/enums.hppandfolder_servicegainsboost/uuid/uuid_io.hpp(include-set sync). - 18 untracked files materialized: the
*_history_provider_registrarfamily ({hpp,cpp} x 9 entities) under core messaging — the family postdates the component's committed tree, the marketdata-class known starting point. The registrar fan-out itself is wired (hand-wired blocks forgmm_componentand the other two config entities, generated sub-registrars for the six remaining entities, and the custompublish_from_dqhandler), so step 7's composition check for the new history providers lands onregistrar.cpp(refdata shape). - Disposal per regen-byproduct-hygiene: the 22 tracked files were
reverted with
git checkout --; the 18 untracked byproducts were disposed with a tagged stash push and immediate drop: tagbyproducts-synthetic-baseline-20260907, entry SHA 8e6ba3b69a9172af4e9c3d39dbdad034ea7f97d5, recoverable via the stash reflog until garbage collection. The tree is clean. - Survey consequence: the per-entity zero-additional-diff rule runs against this baseline; each entity's survey classifies whether its committed output is feature-restorable (bind to byte-identity) or template-era (the current-template shape lands as the deliberate delta).
5.7. Survey classification (2026-09-07, phase 1 close)
All 9 files under projects/ores.synthetic/modeling are
ores.codegen.entity: no field groups, modules, or components to
exclude. Two survey facts ground the matching: the SQL
change_reason_code column and validation are unconditional
provenance machinery on versioned tenant tables (process_parameter_value
declares no change-reason feature and its committed SQL still carries
it), so has_change_reason_cache gates only the client-side cache;
and synthetic has no eventing integration tests on main today, so the
uuid-identified-lookup Physical-space enablement materializes them as
a deliberate addition (the compute/iam precedent).
| Entity | Profile candidate | Verdict |
|---|---|---|
yield_curve_process_type |
simple-lookup |
MATCH: all 7 assignments agree (text code PK, tenant-only, has_uuid_primary_key: false declared, change reason and pagination true). Extras stay per-entity: has_batch_remove, display_order, the tenant_scope: system validation, generator facet. |
yield_curve_process_parameter_definition |
uuid-surrogate-lookup |
MATCH: tenant-only, explicit has_workspace_id: false, uuid PK, pagination; no relationships and no changed event, so the surrogate profile (no Physical-space eventing enablement) fits over uuid-identified-lookup. Phase-3 check: the profile's has_change_reason_cache: true may add cache plumbing to its committed Qt controller if absent. system_tenant_validation: true stays per-entity. |
fx_spot_generation_config |
uuid-identified-lookup |
MATCH: all 5 assignments agree; changed event and notify trigger are live. The Physical-space enablement materializes its eventing integration test. The tenant_scope: both validation and SQL checks stay explicit; capture D583B57B is verified resolved (see below). |
gmm_component |
uuid-identified-lookup |
MATCH: same shape. |
ir_curve_generation_config |
uuid-identified-lookup |
MATCH: same shape; the 5-part natural key stays per-entity. |
ir_curve_template_entry |
uuid-identified-lookup |
MATCH: same shape. |
market_data_generation_config |
uuid-identified-lookup |
MATCH with records: nullable_tenant_id: true and the nullable party_id stay explicit (system-scope configs; not profile-covered), and the scope / binding_mode domain-enum columns stay out of the Qt drawer per the feed_binding / trade_type enum-gap precedent. |
ir_curve_generation_config_process_parameter_value |
NO-MATCH | fk-scoped-child conflicts: it fixes has_workspace_id: true against the org's explicit false, and fixes has_explorer_api: true with none declared. Actual features recorded: uuid PK, tenant-only, has_workspace_id: false, parent FKs to ir_curve_generation_config and yield_curve_process_parameter_definition (the latter :use_system_tenant: true), no Qt, no change-reason cache feature, no changed event. Org stays unbound with explicit properties, per the business_unit precedent. |
folder |
self-referencing-hierarchy |
MATCH: tenant, has_parent_id (with prevent_cycle), uuid PK, workspace false. The Qt half is inert (no drawer; the iam account_contact_information precedent). parent_id_column and hierarchy_name_field stay per-entity, as does the collection_id soft FK. |
Deliberate deltas expected at phase 3, each reviewed when it lands:
- The eventing integration tests for the five uuid-identified-lookup entities materialize (new files; compute/iam precedent).
- The
*_history_provider_registrarfamily (18 files) commits with the bind; skill step 7 composes them intoregistrar.cpp. - The three Qt controllers land the
wireDetailDialogCommontemplate-era shape as their deliberate delta. - The
:profile:declarations replace the covered explicit feature properties; uncovered declarations (has_batch_remove,nullable_tenant_id,system_tenant_validation,tenant_scope, hierarchy fields, generator snippets, Qt drawer keys) stay.
Capture disposition: codegen_sql_validation_fn_drift_synthetic
(D583B57B) is verified resolved by this baseline — the model's
:tenant_scope: both now generates the system-tenant canonical-set
fallback, and the probe reproduced
synthetic_fx_spot_generation_configs_create.sql byte-identically,
fallback included. The capture can close as superseded with this
evidence.
6. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario, run
through the QA Validation Runner panel) that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
7. PRs
8. Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 5576511572 (1) | Unused changeReasonCache_ member on the read-only parameter-definition controller | ores.qt controller templates | Accepted, fixed at source | The controller templates now gate the cache wiring on wires_change_reason_cache (has_change_reason_cache and not has_readonly_paginated_list, computed in core.py); the cascade is contained (no other readonly org declares the cache) and non-readonly controllers render byte-identically. 8f60adbae7. |
| 5576511572 (2) | party_id preset dropped from the Qt mdc create path | MarketDataGenerationConfigDetailDialog.cpp | Confirmed intentional | stamp() derives tenant_id and party_id from the validated JWT context server-side (security boundary); the client-side preset was never authoritative - new configs are stamped with the creating session's party. |
| 5576511572 (3) | RLS party-isolation narrowed to for select | synthetic_rls_policies_create.sql | Confirmed intentional | The refdata pattern (multi_party.org, the rls_party_isolation feature): writes are the trigger set's job; recorded in the RLS commit and the task doc. |
| 5576511572 (4) | Eventing tests cover 7 of 9 entities | core/tests | Confirmed deliberate | parameter_definition is uuid-surrogate-lookup (no Physical-space eventing enablement, no changed event); process_parameter_value is the recorded NO-MATCH. The seven tested are exactly the entities whose profiles enable the eventing integration test address. |
Legacy sweep (2026-09-08, requested alongside the review): every entity-shaped layer in ores.synthetic (protocol, handler, mapper, entity, repository, service, registrar, history provider, history field mapper, generator, table/json io, table display) is generated and drift-checked; no orphan files exist (all cpp files are in the component file lists, all headers referenced); the remaining hand-written files are genuinely custom flows with no codegen equivalent (organisation generation/publishing, publish-from-dq, the feeds/simulator/vintage/preview handlers, financial_names.hpp seed data, app/host/config). The manual wiring that superseded codegen carried (the hand-wired registrar blocks, the hand-era RLS policies, the old folder hierarchy stack) was removed earlier in this task.
8.1. Component audit fixes (2026-09-08)
The component audit's five recommendations, applied on this branch:
- Stale component overviews. The three
component_overview.orgfiles (api, core, service) still described the pre-codegen component (organisation/catalog generation only). All three were rewritten to describe the nine bound entities, the generated facets, the feeds and template resolver, the feed control plane, and the catalog/organisation generators that remain hand-written. - CRUD and history coverage in the integration tests. The shared
eventing integration test template now emits a CRUD round trip
after the eventing assertions: update through the repository, read
the version history through the service, delete, then assert the
row is gone from latest reads while the history keeps every
version. Three template bugs were fixed at source while landing
it: the section now reuses the test's party-scoped context (the
first cut called
set_partywith the SQL-resolving overload, which cannot resolve a generator-stamped party); entities whose party is a parties FK (parent_is_party) build the session's visible-party set from the row's own seeded party; and the read-by-id call honoursservice_find_prefix(find_vsget_). The change propagates to every registry component's eventing tests (84 files: refdata, iam, compute, reporting) — the drift gate is global, so the regeneration is checked in with this PR, and every affected suite runs green locally. - Domain enum converters.
scopeandbinding_modegainedto_string,operator<<, and*_from_string(std::optional) converters, replacing the shell's hand-rolled parsers. - Table template enum streaming. The market data generation
config's Table display declares the
scopeandbinding_modeenum columns; the generated table converter streams them through the newoperator<<. The list shells render the enums. - std::expected migration. Assessed, recorded as a capture, not applied: the ~60 throw sites split into the generated throw/catch-at-handler chain (the codegen pattern across all components — migrating synthetic alone would fork the pattern) and a small set of hand-written feed/resolver throws. A house-wide error-handling convention decision covers both and belongs in its own story.
9. Result
Live-test residue cleanup: the shell_param VASICEK parameter
definition row left in the shared database by the ores.shell session
was soft-deleted; it broke the definitions.size() = 4= catalogue
counts in the feed config handler tests (the test and dev
environments share one database).
Acceptance for this round: full build exit 0; the five affected core suites (refdata, iam, compute, reporting, synthetic) 100% pass with the CRUD round trips in place; roundtrip check exit 0; misspell clean over changed files; cmake component lists regenerated.
Closed 2026-09-08. All nine ores.synthetic entities were surveyed against the profile catalogue; eight bound (yield_curve_process_type to simple-lookup, folder to self-referencing-hierarchy, yield_curve_process_parameter_definition to uuid-surrogate-lookup, and fx_spot_generation_config, gmm_component, ir_curve_generation_config, ir_curve_template_entry and market_data_generation_config to uuid-identified-lookup), one recorded NO-MATCH (ir_curve_generation_config_process_parameter_value: fk-scoped-child conflicts on workspace scope and explorer API). Every bind was verified per entity: zero additional diff against the baseline, and an idempotent second regen. The known starting point (the fx_spot_generation_config validation-fn capture, D583B57B) is verified resolved: the baseline regen reproduced the system-tenant fallback byte-identically; the capture can close as superseded.
The eventing-era history-provider registrar families landed composed into the registrar fan-out (step 7); the drift gate now covers synthetic (step 8): KNOWN_DRIFT_FREE gained the entry and check_component_drift.py –all exits 0 on a clean tree.
Acceptance: every entity surveyed and every match bound; zero additional diff per entity with idempotent regen; the no-match recorded with actual feature values; pre-existing drift fixed at source (the era-sync sign-offs, the hierarchy-machinery restoration, the RLS policy shape, the generator validity fixes); full build exit 0 and ctest 71/71 (908s); schema current per the recreated 0.0.25 database. The story closes after the post-merge live system test per the 2026-09-07 done-shape decision; the ores.shell system test is the closing step.