Task: Migrate lookup_entity into domain_entity; retire the lookup_entity type/loader/pipeline
Table of Contents
This page documents a task in the Commission ores.qt.dq — full-stack codegen for the DQ component story. It captures the goal, current status, acceptance, and any notes or results.
Goal
Migrate the five DQ lookup_entity models from 819B050A onto
domain_entity (has_uuid_primary_key implied by the primary key
column's own :type:, not a separate flag), so they generate through
the mature, actively-maintained domain_entity C++/SQL pipeline
instead of lookup_entity's separate, under-built one. Full retirement
of the ores.codegen.lookup_entity type/loader/dispatch is deferred
to a follow-up task covering the remaining 10 lookup_entity models
outside dq (iam/trading/analytics) – this task only needed to
unblock 819B050A.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Commission ores.qt.dq — full-stack codegen for the DQ component |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing – follow-up filed as AEFAE247. |
| Last touched | 2026-07-18 |
Acceptance
[X]Pilot-migrate one DQ entity (catalog) todomain_entity, verify both SQL and C++ generation end to end against the real hand-written table/pre-existing generated files, before committing to the rest.[X]Migrate the remaining four DQ entities (change_reason_category,change_reason,data_domain,subject_area) the same way, each SQL-verified.[ ]~Add has_coding_scheme/has_image_id/has_artefact_insert_fn as domain_entity knobs~–coding_scheme=/=image_idnot needed: they already exist ondomain_entity(domain_entity_to_table_context=/=normalise_sql_table_context, used by dozens of existing refdata entities).has_artefact_insert_fnis genuinely live (sql_schema_artefact_create.mustache) but none of these 5 entities used it, so porting it to domain_entity was out of scope here – see Notes and AEFAE247.[ ]Migrate the remaining 10 lookup_entity models (iam/trading/analytics) – deferred to a follow-up task, not required to unblock 819B050A.[ ]Retireores.codegen.lookup_entity's =#+type:=/loader/dispatch entirely – blocked on the above (still 10 live consumers).
Plan
(Implementation strategy. Written when work starts; key decisions
are distilled into the parent story's * Decisions at close, but the
plan itself stays — it is the historical record of what we did.)
Notes
Discovered while planning the migration that domain_entity is
substantially more capable than expected, changing the task's actual
scope:
coding_scheme=/=image_idare already first-class domain_entity frontmatter fields (#+coding_scheme: none/required/nullable,#+image_id:), wired throughdomain_entity_to_table_context()andnormalise_sql_table_context()into the same SQL template lookup_entity used (sql_schema_create.mustachefor domain_entity vssql_schema_table_create.mustachefor lookup_entity – two different templates, same behaviour). Dozens of real refdata entities (country,currency, …) already exercise this. No new knobs needed.has_artefact_insert_fnis live:sql_schema_artefact_create.mustache(line 43) renders\{\{#entity.has_artefact_insert_fn\}\}, wired in viaget_schema_template_mappings()for lookup_entity/schema-model entities – an earlier grep pass here wrongly called it dead, only having checkedsql_schema_table_create.mustache, not the artefact template too.lei_entity=/=lei_relationship(both still lookup_entity, not touched by this migration) genuinely depend on it to generate their artefact-table insert function. Harmless for this PR –domain_entity_to_table_context()=/ =get_domain_entity_template_mappings()never read the flag at all, so dropping it from these 5 migrated models changes nothing – but AEFAE247 (the follow-up migratinglei_entity=/=lei_relationship) must explicitly preserve/port this behaviour, sincedomain_entityhas no equivalent hook today. Not ported here because none of these 5 entities used it (all hadhas_artefact_insert_fn: false).domain_entitysupportsnatural_keys(plural) as extra human-meaningful columns, but – checked directly insql_schema_create.mustache– they are not part of the actual SQL primary key or exclusion constraint, only separate unique indexes (mirroring howcountry.org'salpha3_code=/=numeric_codenatural keys get their own* Indexesentries, not a composite PK). So migratingsubject_areato domain_entity does not solve 52FE40FA for free, as first hoped –domain_nameis still a plainColumnwith a soft-FK validation, now paired with a compound unique index (:columns: name, domain_name,:unique: true) enforcing the pair's uniqueness for current rows – stronger than the lookup_entity version (no compound enforcement at all) but still short of a true composite primary key/exclusion constraint. 52FE40FA remains open and necessary for the real fix.
Pilot (catalog) surfaced a process trap worth flagging: catalog
(and other DQ entities) already had pre-existing, tracked C++
domain files sitting unwired in the tree (per the story's own
description: "3 entities' generated files already exist from the
badge migration work, just never wired into the build") predating
their demotion to lookup_entity-only. A first attempt to clean up
post-verification with rm -f on the regenerated output
deleted these tracked files rather than just removing untracked
test-generation output – caught immediately via git status showing
D (deleted, tracked) rather than ?? (untracked), restored via
git checkout --. Every regeneration in this task was verified with
git status --short before cleanup afterwards specifically to catch
this.
Each entity's SQL generation was verified via
compass codegen generate --model ... --address ores.sql.schema
(real, non-dry-run), diffed against the hand-written table it
replaces (or the pre-existing generated file, for catalog's C++),
then reverted (git checkout -- or rm as appropriate, per the trap
above) – only the .org models are committed, matching 819B050A's
own verification pattern. None of the five hand-written DQ tables or
pre-existing generated files have actually been swapped over yet –
that remains 819B050A's own remaining acceptance, now unblocked.
Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario, run
through the QA Validation Runner panel) that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
PRs
| PR | Title |
|---|---|
Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | "has_artefact_insert_fn is dead metadata" claim is wrong – sql_schema_artefact_create.mustache actively renders it, and lei_entity/lei_relationship genuinely depend on it | task_migrate-lookup-entity-into-domain-entity.org, task_migrate-remaining-lookup-entities-to-domain-entity.org | Accepted | Earlier grep only checked sql_schema_table_create.mustache, missing the separate artefact template. Harmless for this PR (none of the 5 migrated entities used the flag, and domain_entity has no equivalent hook regardless), but corrected the claim and added an explicit warning to AEFAE247 so it doesn't silently drop lei_entity/lei_relationship's artefact insert function when migrated. |
Result
Migrated all five DQ entities (catalog, change_reason_category,
change_reason, data_domain, subject_area) from
#+type: ores.codegen.lookup_entity to
#+type: ores.codegen.entity (domain_entity), routing them through
the mature, actively-used domain_entity C++/SQL pipeline instead of
lookup_entity's separate, under-built one – no template changes
needed (unlike the abandoned in-place C++ fix attempted during
DDBD522D), since domain_entity already generates correct SQL and C++
for this exact shape (natural-key-only, has_tenant_id: true, no
coding scheme). Each entity's SQL generation was verified directly
against its real hand-written table; catalog's C++ was additionally
verified against pre-existing (unwired, tracked) generated files,
producing a real improvement (proper tenant_id type instead of
std::string, an added entity_type_of() dispatch function,
clock_timestamp() instead of current_timestamp for same-
transaction correctness, security definer set search_path
hardening) – confirming domain_entity's pipeline is not just
equivalent but more mature than lookup_entity's ever was.
subject_area's compound natural key (name + domain_name) is
still not a true composite primary key/exclusion constraint under
domain_entity (52FE40FA remains open and necessary), but now carries
a compound unique index enforcing the pair for current rows –
stronger than the lookup_entity version, which had no such
enforcement at all.
has_coding_scheme=/=has_image_id knobs did not need porting to
domain_entity – they already exist there as
coding_scheme=/=image_id, used by dozens of real refdata entities.
has_artefact_insert_fn is genuinely live
(sql_schema_artefact_create.mustache, wired via
get_schema_template_mappings()), not dead as first suspected here –
none of these 5 entities used it, so it wasn't ported to domain_entity
in this task, but domain_entity_to_table_context()=/
=get_domain_entity_template_mappings() have no equivalent hook for it
today. Two of the still-unmigrated lookup_entity models
(lei_entity, lei_relationship) genuinely depend on it; AEFAE247
must preserve/port this behaviour explicitly when migrating them, not
assume it can be dropped.
10 lookup_entity models remain outside dq (iam/trading/analytics);
migrating them and retiring ores.codegen.lookup_entity's
=#+type:=/loader/dispatch entirely is filed as a follow-up,
AEFAE247, since it isn't
required to unblock 819B050A.