Task: Bind ores.dq entities to profiles; verify zero-diff regen
Table of Contents
This page documents a task in the Entity classification and drift baseline: ores.dq story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Same method as the pilot task (Bind ores.refdata entities to profiles): survey every ores.dq entity (projects/ores.dq/modeling) against the profile catalogue, rebind matches, regenerate, and require zero additional diff against the pre-rebind checked-in output.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Entity classification and drift baseline: ores.dq |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-08-04 |
3. Acceptance
- Every entity discovered under projects/ores.dq/modeling is surveyed against all Variability Profiles.
- Every matching entity is rebound (profile property set, explicit per-entity properties the profile covers removed) and regenerated with zero additional diff versus its pre-rebind checked-in output.
- No-match entities are recorded with their actual feature values, not silently dropped.
- Any pre-existing drift the regen+diff surfaces (template bug, stale checked-in output) is fixed, not worked around.
- The known dataset_bundle service_find_by_uuid/service_find_by_code gap and the HexPrefix model typo (carried over from the superseded ad hoc task) are resolved as part of reaching zero drift.
4. Plan
(Implementation strategy. Written when work starts; key decisions
are distilled into the parent story's * Decisions at close, but the
plan itself stays — it is the historical record of what we did.)
5. Notes
- Found and fixed a real codegen bug while establishing the pre-rebind
baseline (regen+diff before touching any model, per the pilot method):
load_org_lookup_entity_model()never read the* Primary keyheading every*_lookup_entity.orgmodel in the repo uses to declare its key (:column:/:type:/:is_text: on the heading itself) – it only handled the domain_entity convention of a:primary_key: true-flagged** columnsub-heading under* Columns. Silent effect:entity.primary_key.columnrendered empty insql_schema_table_create.mustache, corrupting the generated DDL (empty column name in CREATE TABLE / PRIMARY KEY / CHECK) for every lookup entity component-wide – not dq-specific. Fixed inprojects/ores.codegen/src/codegen/org_loader.py, regression-tested inprojects/ores.codegen/tests/test_org_loader_lookup_entity.py; confirmed zero-diff forartefact_type(the previously-broken entity) after the fix. Full codegen test suite (110 tests) green. - Baseline check also surfaced pre-existing drift unrelated to the primary- key bug: a stale blank line before a comment in the notify/validation trigger SQL, on 6 tables (change_reasons, code_domains, data_domains, dataset_bundles, subject_areas, change_reason_categories) – same class as the pilot's currency_calendar finding. Not yet fixed; next step.
- A full-component regeneration (triggered incidentally by
check_component_drift.py --components dq, which regenerates every address) additionally surfaced ~130 modified/untracked C++/Qt files acrossores.dq.api,ores.dq.core, andores.qt.dq– the component's C++/Qt layers appear to have substantial pre-existing drift beyond SQL, including 4 entities (lei_entity, lei_relationship, report_definition, synthetic_fx_spot_config) with no checked-in generated code at all yet. Reverted rather than committed blind – this is a large, separate investigation (each new table needs drop.sql, notify trigger, population, validation-ignore entries, and full DB/CMake wiring, not just template output) and deserves its own deliberate pass rather than folding into this survey uninspected. - Root-caused the C++ gap via deep analysis (user-requested) rather than
patching lookup_entity's own C++ path further:
lookup_entityis deliberately being retired, not maintained. Task 464B5EC6 (DONE) already proved and executed the fix – migrate each remaininglookup_entitymodel todomain_entity(has_uuid_primary_key: false), which gets a complete, mature SQL+C++ projection for free. Task AEFAE247 (BACKLOG) is the pre-existing, already-filed follow-up covering exactly dq's 5 remaining lookup entities plus 10 more outside dq. Adopted AEFAE247's plan here instead of patching lookup_entity's C++ generation (the approach DDBD522D already tried and explicitly abandoned). artefact_typemigrated todomain_entity(commit on this branch), following the 464B5EC6/catalog playbook exactly: one address at a time (SQL schema + notify trigger, domain/json_io/table, repository/entity/ mapper, service, protocol, NATS handler/registrar/event-registrar, presentation, generator), each reviewed before the next, idempotency verified (zero additional diff on a second regenerate pass). Repository moved from stateful (ctx in constructor) to the standard stateless (ctx per call) shape;publication_service(its one caller) updated accordingly. Wired intoregistrar.cppalongside catalog/data_domain. Addedrepository_artefact_type_repository_tests.cpp.ores.dq.api.lib=/ =ores.dq.core.lib=/=ores.dq.service.libbuild clean;ores.dq.api.tests(273/50) andores.dq.core.tests(300/95, incl. 7 new cases) pass against the live DB.- Built the
has_artefact_insert_fnport 464B5EC6 flagged as missing, then migratedlei_entity: added a newores.sql.schema.domain_entity_artefact_createarchetype (#+model_types: domain_entity, disabled by default – most domain_entity models have no artefact table at all, an entity opts in with:ores.sql.schema.domain_entity_artefact_create.enabled: truein its file-level PROPERTIES drawer), ported from the lookup_entity-onlysql_schema_artefact_create.mustache; wiredhas_artefact_insert_fnand* Artefact indexesas domain_entity frontmatter/section support in org_loader.py. Regenerated artefact table matches the pre-existing checked-in one exactly bar two harmless deltas (template-name comment; two redundantdefault nullclauses correctly dropped). Idempotent on a second regen. Confirmed the new archetype has zero effect on any other domain_entity (artefact_type and all 5 already-migrated dq entities regenerate byte-identical). Tangle round-trip verified (hand-written.mustachematchescompass build --direct tangle_codegen_templatesoutput exactly). - Discovered along the way:
lei_entity(like the other 3 remaining lookup entities) has never had a "main" bi-temporal table in production –dq_lei_entities_create.sqlnever existed in the repo; LEI data is queried directly from the artefact table (perdq_lei_entities_query_functions_create.sql's own doc comment). Sincedomain_entity's main-table archetype has no per-entity opt-out, it still generates a speculative main table + notify trigger as a byproduct when regenerated at--address ores.sql.schema– correctly left uncommitted (nothing in the codebase would reference it), same treatment given to lei_relationship/report_definition/ synthetic_fx_spot_config once each is migrated. lei_relationshipmigrated using the same hook. Regenerated artefact table matched the checked-in one bar the template-name comment and two artefact-index names: the checked-in file's "start_node_id_idx"/ "rel_type_idx" didn't match what its own original lookup_entity model would ever have produced (no:name:override on either Artefact-index heading) – pre-existing drift (hand-shortened after generation, never regenerated since), not something the migration introduced. Verified no other checked-in SQL referenced the old names before accepting the correct, template-derived ones. Idempotent on a second regen. Same no-main-table treatment aslei_entity.report_definitionmigrated (UUID primary key this time –:type: uuidon theidcolumn is all that's needed,is_uuid=/=cpp_typederived automatically). Surfaced and fixed a real template bug: its two-line prose body broke the artefact template's single-line SQL comment (embedded newline exits the comment mid-line into bare SQL). Addeddomain_entity.description_oneline(whitespace-joined) alongside the existingdescription_formatted, pointed the artefact template at it; verified zero effect on lei_entity/lei_relationship (already single-line) or any other domain_entity. Tangle round-trip re-verified. Regenerated artefact table byte-identical to checked-in bar the template-name comment. Idempotent.synthetic_fx_spot_configmigrated – last of the 4. Boolean/double precision column types and an auto-quoted text default (price_sourcevintage) all rendered correctly through the domain_entity column pipeline unchanged. Regenerated artefact table matched the checked-in one bar the template-name comment and the header text: the checked-in file's comment silently truncated the model's two-sentence body to its first sentence – same stale-checked-in-file drift class seen throughout this task, not a template bug. Idempotent. Verified zero effect on the other 3 migrated entities or any other domain_entity.grep -rl "ores.codegen.lookup_entity" projects/ores.dq/modeling/*.orgnow returns zero live models (only a stale prose mention in thepublication_pipeline.orgknowledge doc, describing the now-retired approach – worth a follow-up doc fix, not blocking this task). All 4 of dq's lookup_entity models are migrated to domain_entity.Profile-binding survey run across every
ores.dqentity/junction (17 files: 13domain_entity, 2junction, 1field_groupexcluded – structurally not an entity, matching the pilot's own precedent for non-entity metatypes). Note on method: none of dq's entities have a** Qtdrawer exceptbadge_definition=/=code_domain=/=dataset_bundle, so for the other 10 theqt-namespace half of every profile's Assignments is inert (nothing reads it, since Qt generation is gated by the drawer's presence, not by profile flags) – match/no-match there turns on theroot=/=sql-namespace features alone (has_tenant_id,has_workspace_id,has_parent_id).* Bound (9) – see Result for the commit
Entity/junction Profile catalog simple-lookup change_reason_category simple-lookup change_reason simple-lookup data_domain simple-lookup artefact_type simple-lookup lei_entity simple-lookup lei_relationship simple-lookup badge_mapping_junction tenant-scoped-junction dataset_bundle_member_junction tenant-scoped-junction * No-match (6) – real data, not a failure; each recorded with its reason
Entity Reason subject_area Compound primary key (name + domain_name, both flagged :primary_key: true) – no named profile covers a compound-key shape; matches 464B5EC6's own note that subject_area's key situation (52FE40FA) remains open.report_definition UUID surrogate key, no Qt, no parent/hierarchy relation. No profile fits: simple-lookup/fully-featured-lookup require a text natural key; self-referencing-hierarchy/fk-scoped-child require has_parent_id/parent_entity_singular, neither applicable. synthetic_fx_spot_config Same shape as report_definition (UUID PK, no Qt, no hierarchy) – same no-match reasoning. dataset_bundle UUID PK with Qt (has_uuid_primary_key: true already correctly set), but not hierarchical/parent-scoped – no profile's Purpose matches an editable, non-hierarchical UUID-keyed entity; forcing self-referencing-hierarchy or fk-scoped-child purely on flag proximity would be wrong the same way business_unit was correctly left unbound in the pilot. badge_definition Qt present, has_uuid_primary_key: false and has_change_reason_cache: true already match simple-lookup, but has_pagination is unset (defaults false) where the profile fixes true. Not treated as an obvious gap like the pilot's leg_type (missing audit trail) – pagination is a UI/product decision, not a data-integrity omission; recorded as a real, deliberate-looking mismatch rather than force-added. code_domain Qt present, has_uuid_primary_key: false matches simple-lookup, but both has_change_reason_cache and has_pagination are unset (default false) against the profile's true/true. Same reasoning as badge_definition – a real mismatch, not silently forced. Both Qt gaps (badge_definition, code_domain) are worth a follow-up product conversation (add pagination/change-reason-cache, or the profile genuinely doesn't fit and that's fine) – out of scope to decide unilaterally here.
6. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario, run
through the QA Validation Runner panel) that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
| Verify ores.dq commissioned entities | FAILED | CRUD+history on catalog/data_domain/change_reason/change_reason_category/artefact_type/dataset_bundle; read-only browse on lei_entity/lei_relationship; confirms report_definition/synthetic_fx_spot_config have no Qt UI; two-client eventing check. Found 3 bugs (artefact_type missing fields + no eventing, change_reason/change_reason_category duplicates), all fixed on this PR before merge. |
| Verify ores.dq PR #1898 bug fixes | PASSED | Retest of the 3 bugs found above, plus the steps left PENDING last round (dataset_bundle CRUD, LEI registry read-only, report_definition/synthetic_fx_spot_config absence). All 8 steps passed. |
| Verify PR #1915's shared-tenant edit fix | PENDING | Edits a seeded row of all 10 tables PR #1915 extended the per-tenant-copy fix to (data_domain, subject_area, methodology, origin/nature/treatment_dimension, coding_scheme, coding_scheme_authority_type, badge_severity, badge_definition); confirms none produce a duplicate row. |
7. PRs
| PR | Title |
|---|---|
| #1922 | [agile] Fill in LEI step's Result |
| #1920 | [agile] Add retest scenario for PR #1915's fix |
| #1915 | [dq,qt,sql] Fix shared-tenant edit duplicates; remove broken LEI browsing UI |
| #1910 | [agile] Add retest scenario for PR #1898's bug fixes |
| #1898 | [dq] Bind ores.dq entities to profiles; commission full C++/Qt stack |
8. Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | Dangling org-link to deleted ores.qt.data_management (site build failing) | 5 modeling docs | Accepted | Repointed all id-links to the deleted component onto ores.qt.dq; verified compass build --direct site succeeds |
| 2 | ores.qt.dq/modeling/component_overview.org still describes only badges, template placeholders unfilled | projects/ores.qt/dq/modeling/component_overview.org | Accepted | Rewrote Summary/Inputs/Outputs/Entry points/Dependencies/See also to reflect the full consolidated scope |
| 3 | artefact_type roundtrip test only checks code/description, not the fields this PR fixed | projects/ores.dq/core/tests/repository_artefact_type_repository_tests.cpp | Accepted | Added CHECKs for artefact_table/target_table/target_subject/display_order |
| 4 | PR #1915 fixes only 2 of ~13 tables sharing the same duplicate-on-edit RLS pattern | projects/ores.sql/create/dq/dq_rls_policies_create.sql, iam_tenant_provisioner_create.sql | Accepted | Audited all flagged tables for the two true preconditions (DDL-seeded + Qt-editable); extended the same per-tenant-copy + RLS fix to 10 confirmed positives (data_domains, subject_areas, coding_scheme_authority_types, coding_schemes, origin/nature/treatment_dimensions, methodologies, badge_severities, badge_definitions). Deliberately excluded catalogs (not seeded), datasets (7-table FK chain, too risky to rush), and badge_mappings/dataset_bundle_members (read-only in the UI, not reachable) |
| 5 | New artefact_types RLS policies missing matching drop statements | projects/ores.sql/drop/dq/dq_rls_policies_drop.sql | Accepted | Added the missing drop policy pair |
| 6 | Stale "LEI Registry" menu references in 2 component-overview docs | projects/ores.qt/dq/modeling/component_overview.org, projects/ores.qt/application/modeling/component_overview.org | Accepted | Removed all 4 references |
9. Result
Bound ores.dq SQL entities to variability profiles with verified zero-diff regen. Commissioned full C++/Qt for lei_entity, lei_relationship, report_definition, and synthetic_fx_spot_config. Consolidated every ores.dq-owned entity's Qt layer into ores.qt.dq, decommissioning the ores.qt.data_management stepping-stone plugin entirely. Migrated catalog/change_reason/change_reason_category/data_domain/dataset_bundle onto full codegen with the generic HistoryDialog; migrated subject_area's history dialog by hand (compound-key exception). Found and fixed several bugs surfaced by end-to-end GUI test scenarios: artefact_type's missing Detail fields and its never-firing NOTIFY trigger (same gap fixed for code_domain and dataset_bundle), and a duplicate-row RLS bug on change_reason/change_reason_category caused by a redundant system-tenant read-policy fallback on tables the tenant provisioner already copies per-tenant. All fixes verified via direct SQL queries against a freshly provisioned Acme Corporation tenant and compass build rat. PR #1898 merged after one review round (dangling org-link to the deleted component, stale component_overview.org, thin test coverage – all addressed).