Task: Bind ores.dq entities to profiles; verify zero-diff regen
Table of Contents
This page documents a task in the Entity classification and drift baseline: ores.dq story. It captures the goal, current status, acceptance, and any notes or results.
Goal
Same method as the pilot task (Bind ores.refdata entities to profiles): survey every ores.dq entity (projects/ores.dq/modeling) against the profile catalogue, rebind matches, regenerate, and require zero additional diff against the pre-rebind checked-in output.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Entity classification and drift baseline: ores.dq |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-08-04 |
Acceptance
- Every entity discovered under projects/ores.dq/modeling is surveyed against all Variability Profiles.
- Every matching entity is rebound (profile property set, explicit per-entity properties the profile covers removed) and regenerated with zero additional diff versus its pre-rebind checked-in output.
- No-match entities are recorded with their actual feature values, not silently dropped.
- Any pre-existing drift the regen+diff surfaces (template bug, stale checked-in output) is fixed, not worked around.
- The known dataset_bundle service_find_by_uuid/service_find_by_code gap and the HexPrefix model typo (carried over from the superseded ad hoc task) are resolved as part of reaching zero drift.
Plan
(Implementation strategy. Written when work starts; key decisions
are distilled into the parent story's * Decisions at close, but the
plan itself stays — it is the historical record of what we did.)
Notes
- Found and fixed a real codegen bug while establishing the pre-rebind
baseline (regen+diff before touching any model, per the pilot method):
load_org_lookup_entity_model()never read the* Primary keyheading every*_lookup_entity.orgmodel in the repo uses to declare its key (:column:/:type:/:is_text: on the heading itself) – it only handled the domain_entity convention of a:primary_key: true-flagged** columnsub-heading under* Columns. Silent effect:entity.primary_key.columnrendered empty insql_schema_table_create.mustache, corrupting the generated DDL (empty column name in CREATE TABLE / PRIMARY KEY / CHECK) for every lookup entity component-wide – not dq-specific. Fixed inprojects/ores.codegen/src/codegen/org_loader.py, regression-tested inprojects/ores.codegen/tests/test_org_loader_lookup_entity.py; confirmed zero-diff forartefact_type(the previously-broken entity) after the fix. Full codegen test suite (110 tests) green. - Baseline check also surfaced pre-existing drift unrelated to the primary- key bug: a stale blank line before a comment in the notify/validation trigger SQL, on 6 tables (change_reasons, code_domains, data_domains, dataset_bundles, subject_areas, change_reason_categories) – same class as the pilot's currency_calendar finding. Not yet fixed; next step.
- A full-component regeneration (triggered incidentally by
check_component_drift.py --components dq, which regenerates every address) additionally surfaced ~130 modified/untracked C++/Qt files acrossores.dq.api,ores.dq.core, andores.qt.dq– the component's C++/Qt layers appear to have substantial pre-existing drift beyond SQL, including 4 entities (lei_entity, lei_relationship, report_definition, synthetic_fx_spot_config) with no checked-in generated code at all yet. Reverted rather than committed blind – this is a large, separate investigation (each new table needs drop.sql, notify trigger, population, validation-ignore entries, and full DB/CMake wiring, not just template output) and deserves its own deliberate pass rather than folding into this survey uninspected. - Root-caused the C++ gap via deep analysis (user-requested) rather than
patching lookup_entity's own C++ path further:
lookup_entityis deliberately being retired, not maintained. Task 464B5EC6 (DONE) already proved and executed the fix – migrate each remaininglookup_entitymodel todomain_entity(has_uuid_primary_key: false), which gets a complete, mature SQL+C++ projection for free. Task AEFAE247 (BACKLOG) is the pre-existing, already-filed follow-up covering exactly dq's 5 remaining lookup entities plus 10 more outside dq. Adopted AEFAE247's plan here instead of patching lookup_entity's C++ generation (the approach DDBD522D already tried and explicitly abandoned). artefact_typemigrated todomain_entity(commit on this branch), following the 464B5EC6/catalog playbook exactly: one address at a time (SQL schema + notify trigger, domain/json_io/table, repository/entity/ mapper, service, protocol, NATS handler/registrar/event-registrar, presentation, generator), each reviewed before the next, idempotency verified (zero additional diff on a second regenerate pass). Repository moved from stateful (ctx in constructor) to the standard stateless (ctx per call) shape;publication_service(its one caller) updated accordingly. Wired intoregistrar.cppalongside catalog/data_domain. Addedrepository_artefact_type_repository_tests.cpp.ores.dq.api.lib=/ =ores.dq.core.lib=/=ores.dq.service.libbuild clean;ores.dq.api.tests(273/50) andores.dq.core.tests(300/95, incl. 7 new cases) pass against the live DB.- Built the
has_artefact_insert_fnport 464B5EC6 flagged as missing, then migratedlei_entity: added a newores.sql.schema.domain_entity_artefact_createarchetype (#+model_types: domain_entity, disabled by default – most domain_entity models have no artefact table at all, an entity opts in with:ores.sql.schema.domain_entity_artefact_create.enabled: truein its file-level PROPERTIES drawer), ported from the lookup_entity-onlysql_schema_artefact_create.mustache; wiredhas_artefact_insert_fnand* Artefact indexesas domain_entity frontmatter/section support in org_loader.py. Regenerated artefact table matches the pre-existing checked-in one exactly bar two harmless deltas (template-name comment; two redundantdefault nullclauses correctly dropped). Idempotent on a second regen. Confirmed the new archetype has zero effect on any other domain_entity (artefact_type and all 5 already-migrated dq entities regenerate byte-identical). Tangle round-trip verified (hand-written.mustachematchescompass build --direct tangle_codegen_templatesoutput exactly). - Discovered along the way:
lei_entity(like the other 3 remaining lookup entities) has never had a "main" bi-temporal table in production –dq_lei_entities_create.sqlnever existed in the repo; LEI data is queried directly from the artefact table (perdq_lei_entities_query_functions_create.sql's own doc comment). Sincedomain_entity's main-table archetype has no per-entity opt-out, it still generates a speculative main table + notify trigger as a byproduct when regenerated at--address ores.sql.schema– correctly left uncommitted (nothing in the codebase would reference it), same treatment given to lei_relationship/report_definition/ synthetic_fx_spot_config once each is migrated. lei_relationshipmigrated using the same hook. Regenerated artefact table matched the checked-in one bar the template-name comment and two artefact-index names: the checked-in file's "start_node_id_idx"/ "rel_type_idx" didn't match what its own original lookup_entity model would ever have produced (no:name:override on either Artefact-index heading) – pre-existing drift (hand-shortened after generation, never regenerated since), not something the migration introduced. Verified no other checked-in SQL referenced the old names before accepting the correct, template-derived ones. Idempotent on a second regen. Same no-main-table treatment aslei_entity.report_definitionmigrated (UUID primary key this time –:type: uuidon theidcolumn is all that's needed,is_uuid=/=cpp_typederived automatically). Surfaced and fixed a real template bug: its two-line prose body broke the artefact template's single-line SQL comment (embedded newline exits the comment mid-line into bare SQL). Addeddomain_entity.description_oneline(whitespace-joined) alongside the existingdescription_formatted, pointed the artefact template at it; verified zero effect on lei_entity/lei_relationship (already single-line) or any other domain_entity. Tangle round-trip re-verified. Regenerated artefact table byte-identical to checked-in bar the template-name comment. Idempotent.synthetic_fx_spot_configmigrated – last of the 4. Boolean/double precision column types and an auto-quoted text default (price_sourcevintage) all rendered correctly through the domain_entity column pipeline unchanged. Regenerated artefact table matched the checked-in one bar the template-name comment and the header text: the checked-in file's comment silently truncated the model's two-sentence body to its first sentence – same stale-checked-in-file drift class seen throughout this task, not a template bug. Idempotent. Verified zero effect on the other 3 migrated entities or any other domain_entity.grep -rl "ores.codegen.lookup_entity" projects/ores.dq/modeling/*.orgnow returns zero live models (only a stale prose mention in thepublication_pipeline.orgknowledge doc, describing the now-retired approach – worth a follow-up doc fix, not blocking this task). All 4 of dq's lookup_entity models are migrated to domain_entity.Profile-binding survey run across every
ores.dqentity/junction (17 files: 13domain_entity, 2junction, 1field_groupexcluded – structurally not an entity, matching the pilot's own precedent for non-entity metatypes). Note on method: none of dq's entities have a** Qtdrawer exceptbadge_definition=/=code_domain=/=dataset_bundle, so for the other 10 theqt-namespace half of every profile's Assignments is inert (nothing reads it, since Qt generation is gated by the drawer's presence, not by profile flags) – match/no-match there turns on theroot=/=sql-namespace features alone (has_tenant_id,has_workspace_id,has_parent_id).* Bound (9) – see Result for the commit
Entity/junction Profile catalog simple-lookup change_reason_category simple-lookup change_reason simple-lookup data_domain simple-lookup artefact_type simple-lookup lei_entity simple-lookup lei_relationship simple-lookup badge_mapping_junction tenant-scoped-junction dataset_bundle_member_junction tenant-scoped-junction * No-match (6) – real data, not a failure; each recorded with its reason
Entity Reason subject_area Compound primary key (name + domain_name, both flagged :primary_key: true) – no named profile covers a compound-key shape; matches 464B5EC6's own note that subject_area's key situation (52FE40FA) remains open.report_definition UUID surrogate key, no Qt, no parent/hierarchy relation. No profile fits: simple-lookup/fully-featured-lookup require a text natural key; self-referencing-hierarchy/fk-scoped-child require has_parent_id/parent_entity_singular, neither applicable. synthetic_fx_spot_config Same shape as report_definition (UUID PK, no Qt, no hierarchy) – same no-match reasoning. dataset_bundle UUID PK with Qt (has_uuid_primary_key: true already correctly set), but not hierarchical/parent-scoped – no profile's Purpose matches an editable, non-hierarchical UUID-keyed entity; forcing self-referencing-hierarchy or fk-scoped-child purely on flag proximity would be wrong the same way business_unit was correctly left unbound in the pilot. badge_definition Qt present, has_uuid_primary_key: false and has_change_reason_cache: true already match simple-lookup, but has_pagination is unset (defaults false) where the profile fixes true. Not treated as an obvious gap like the pilot's leg_type (missing audit trail) – pagination is a UI/product decision, not a data-integrity omission; recorded as a real, deliberate-looking mismatch rather than force-added. code_domain Qt present, has_uuid_primary_key: false matches simple-lookup, but both has_change_reason_cache and has_pagination are unset (default false) against the profile's true/true. Same reasoning as badge_definition – a real mismatch, not silently forced. Both Qt gaps (badge_definition, code_domain) are worth a follow-up product conversation (add pagination/change-reason-cache, or the profile genuinely doesn't fit and that's fine) – out of scope to decide unilaterally here.
Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario, run
through the QA Validation Runner panel) that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
| Verify ores.dq commissioned entities | FAILED | CRUD+history on catalog/data_domain/change_reason/change_reason_category/artefact_type/dataset_bundle; read-only browse on lei_entity/lei_relationship; confirms report_definition/synthetic_fx_spot_config have no Qt UI; two-client eventing check. Found 3 bugs (artefact_type missing fields + no eventing, change_reason/change_reason_category duplicates), all fixed on this PR before merge. |
| Verify ores.dq PR #1898 bug fixes | PASSED | Retest of the 3 bugs found above, plus the steps left PENDING last round (dataset_bundle CRUD, LEI registry read-only, report_definition/synthetic_fx_spot_config absence). All 8 steps passed. |
| Verify PR #1915's shared-tenant edit fix | PENDING | Edits a seeded row of all 10 tables PR #1915 extended the per-tenant-copy fix to (data_domain, subject_area, methodology, origin/nature/treatment_dimension, coding_scheme, coding_scheme_authority_type, badge_severity, badge_definition); confirms none produce a duplicate row. |
PRs
| PR | Title |
|---|---|
| #1922 | [agile] Fill in LEI step's Result |
| #1920 | [agile] Add retest scenario for PR #1915's fix |
| #1915 | [dq,qt,sql] Fix shared-tenant edit duplicates; remove broken LEI browsing UI |
| #1910 | [agile] Add retest scenario for PR #1898's bug fixes |
| #1898 | [dq] Bind ores.dq entities to profiles; commission full C++/Qt stack |
Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | Dangling org-link to deleted ores.qt.data_management (site build failing) | 5 modeling docs | Accepted | Repointed all id-links to the deleted component onto ores.qt.dq; verified compass build --direct site succeeds |
| 2 | ores.qt.dq/modeling/component_overview.org still describes only badges, template placeholders unfilled | projects/ores.qt/dq/modeling/component_overview.org | Accepted | Rewrote Summary/Inputs/Outputs/Entry points/Dependencies/See also to reflect the full consolidated scope |
| 3 | artefact_type roundtrip test only checks code/description, not the fields this PR fixed | projects/ores.dq/core/tests/repository_artefact_type_repository_tests.cpp | Accepted | Added CHECKs for artefact_table/target_table/target_subject/display_order |
| 4 | PR #1915 fixes only 2 of ~13 tables sharing the same duplicate-on-edit RLS pattern | projects/ores.sql/create/dq/dq_rls_policies_create.sql, iam_tenant_provisioner_create.sql | Accepted | Audited all flagged tables for the two true preconditions (DDL-seeded + Qt-editable); extended the same per-tenant-copy + RLS fix to 10 confirmed positives (data_domains, subject_areas, coding_scheme_authority_types, coding_schemes, origin/nature/treatment_dimensions, methodologies, badge_severities, badge_definitions). Deliberately excluded catalogs (not seeded), datasets (7-table FK chain, too risky to rush), and badge_mappings/dataset_bundle_members (read-only in the UI, not reachable) |
| 5 | New artefact_types RLS policies missing matching drop statements | projects/ores.sql/drop/dq/dq_rls_policies_drop.sql | Accepted | Added the missing drop policy pair |
| 6 | Stale "LEI Registry" menu references in 2 component-overview docs | projects/ores.qt/dq/modeling/component_overview.org, projects/ores.qt/application/modeling/component_overview.org | Accepted | Removed all 4 references |
Result
Bound ores.dq SQL entities to variability profiles with verified zero-diff regen. Commissioned full C++/Qt for lei_entity, lei_relationship, report_definition, and synthetic_fx_spot_config. Consolidated every ores.dq-owned entity's Qt layer into ores.qt.dq, decommissioning the ores.qt.data_management stepping-stone plugin entirely. Migrated catalog/change_reason/change_reason_category/data_domain/dataset_bundle onto full codegen with the generic HistoryDialog; migrated subject_area's history dialog by hand (compound-key exception). Found and fixed several bugs surfaced by end-to-end GUI test scenarios: artefact_type's missing Detail fields and its never-firing NOTIFY trigger (same gap fixed for code_domain and dataset_bundle), and a duplicate-row RLS bug on change_reason/change_reason_category caused by a redundant system-tenant read-policy fallback on tables the tenant provisioner already copies per-tenant. All fixes verified via direct SQL queries against a freshly provisioned Acme Corporation tenant and compass build rat. PR #1898 merged after one review round (dangling org-link to the deleted component, stale component_overview.org, thin test coverage – all addressed).