Task: Model the handcrafted ores.dq code and generate the artefact tables

Every ores.dq table, view and SQL function has a model, or a recorded reason for having none. Regeneration stays byte-identical, so the tree stops carrying handcrafted SQL that no model owns.

This is pass 1 of two. Pass 2 runs the Component Clean Standard over the tree this task leaves behind.

The design question is already answered: an artefact table can generate from the same entity model as its base table, and 19 of the 49 artefact tables already do. See the investigation.

• Every table in projects/ores.sql/create/dq is generated from a model, or recorded with the reason it is not.
• Every handcrafted *_artefact_create.sql file is either generated from a model or deleted.
• The publication and preview functions generate from a template, or the dead template is deleted with the reason recorded.
• The two shadowed table pairs and their drop scripts are collapsed to one owner each.
• check_component_drift.py --component dq --dry-run reports no drift, twice in a row.
• Every generator defect found is fixed at its source with a regression test in the codegen suite.
• No table is deleted without a consumer census recorded on this task.

Work in verifiable units. Run the census and the drift check after each unit, and record the result here.

1. Baseline. Re-run survey_sql_artefacts.py, survey_component.py and the drift check. Keep the three outputs on this task.
2. Liveness census. For each of the 7 orphan tables and the 13 generated tables with no model, list every reader and every writer: populate scripts, publish functions, RLS policies, drop scripts and C++ consumers. Decide keep, model, or delete. Record one row per table.
3. Reconcile the 12 divergent tables. For each, decide whether the artefact table or the projection is wrong. Record the decision and the reason.
4. Extend the 6 foreign models. Add the artefact facet in the owning component. Do not create a DQ model.
5. Switch the 5 derivable tables to the archetype. Delete the handcrafted twins.
6. Collapse the two shadowed pairs. Keep the generated file, rewire dq_create.sql and dq_drop.sql, delete the handcrafted twin.
7. Decide the publication-function template. Either wire sql_populate_function_refdata.mustache into the ores.sql.populate facet and absorb the 21 handcrafted files, or delete the template and record why. This is the largest automation gain, so take it before the mechanical steps if time is short.
8. Author the remaining models: the 10 handcrafted notify triggers, the FSM family, publication, bundle_publication, rls_policies and functions.
9. Regenerate the C++ tree and reach byte-identical.
10. Resolve the 21 hand-written eventing headers (*_changed_event.hpp). G01 treats eventing as generatable.

Run on 2026-09-26. Four facets are now byte-identical and committed; the fifth is not mechanical, and this section records why.

Four facets landed. SQL (11 files), cmake (1), doc (14 recipes plus their tangled scripts), ts.protocol (1). The SQL facet carried the version-conflict guard and the canonical event payload; the cmake facet the generated marker; the doc facet the fourteen shell recipes dq's entities never had.

The C++ facet is three changes at once. Its dry run is 480 files, and applying it produced 18 compile errors. Reading them:

So the facet performs the canonical NATS entity protocol migration (P01) for all fourteen entities, removes the hand-written umbrellas (G02), and rewires the registrar's composition points (W01) — and the last two only make sense entity by entity. The clean standard says so directly: "Work one entity at a time. Regenerate, build and test after each one."

The blanket apply was reverted, not committed. A half-migrated tree with 480 uncommitted files is worse than a clean one. The findings above are the round's real product.

artefact_type is migrated and builds. 29 files regenerated, four created (the event header, the history provider registrar pair, and the eventing integration test), four component_files.cmake lists refreshed. Its protocol now states list, get, get_many, put, delete, history and the point-in-time read with the key, write, change, removal, lookup, event, version_key and versions_filter types. Nothing outside ores.dq consumes its protocol, so the migration is self-contained.

One hand-written fix was needed on the way: dq's core tests target did not link ores.eventing.api.lib or ores.eventing.core.lib, and the generated eventing integration test includes a header from the latter. ores.synthetic's equivalent target links both, so this is a gap in dq's CMakeLists rather than a template defect. The suite itself needs a live database, which this sandbox does not have, so the build and link are the available evidence.

Two findings recorded.

• projects/ores.shell/dq is a directory that is not a catalogue component and is not added by ores.shell/CMakeLists.txt, unlike ores.shell.analytics, .assets, .iam, .refdata, .synthetic and .trading. The shell command units the facet generates for dq are therefore compiled by nothing. ores.shell.workflow has the same shape, and validate_docs.sh already reports it.
• The umbrella headers and the bespoke handlers they carry (badge_handler, badge_service, data_organization_handler, dimension_handler) are consumed only by registrar.cpp, so they can go. But each carries registrations for entities that are not modelled yet — methodologies, dimensions, datasets, coding schemes, publications — and those have to move before the umbrella can be deleted.

Figures. Codegen suite 686 passing; validator 0 warnings; dq's component_files lists up to date; ores.dq.core.tests builds and links.

Run on 2026-09-27. The one dq test failure part twenty-two left undiagnosed is now understood and fixed.

The drawer was transcribed incompletely. When the coding scheme pair was modelled, ** Table display was written with six columns. The hand-written renderer it replaced had eight: Code, Name, Authority Type, Subject Area, Domain, Description, Modified By, Version. The model dropped subject_area_name and description, and shortened the authority_type header to Authority. The generated table streamed six fields, and domain_coding_scheme_tests.cpp – which asserts the rendered table contains Description and the description's own value – was the only thing that noticed.

Why nothing else caught it. The commit that modelled the pair reports that "the generated tables match the hand-written ones column for column", and that was true of the SQL tables. The display drawer is a separate list with no gate behind it, so a column can leave it silently. The census that bounds the damage: five dq models declare a description column, and coding_scheme is the only one whose hand-written table displayed it.

The fix. The drawer is restated with all eight columns in the hand-written order, so the regenerated renderer is behaviourally identical to the one the model replaced. compass codegen generate --model projects/ores.dq/modeling/ores.dq.coding_scheme.org --address ores.cpp rewrote exactly one file, coding_scheme_table.cpp, which is the whole of the diff.

Evidence. ores.dq.api.tests passes 50 of 50, up from 49. ores.dq.core.tests is unchanged at 109 of 111, its two failures being the LEI cases part twenty-two gave their own task; that task has since landed as PR #2270, so the suite now passes. ores.dq.service.tests and ores.shell.dq.tests pass. The codegen suite passes 845. dq has no failing test case that this task owns.

The wider sweep, for the record. All 83 test binaries were run against a freshly recreated database, first run. Six are red, and five are not dq: refdata core (11 cases) and marketdata core (2) both follow from 48f4282df6, which made a version-zero create strict and moved the refdata notify payload from id to short_code; scheduler core (1) and shell api (1) are tests added on 2026-09-26 that fail on their own subjects. None of the five is caused by this task.

Run on 2026-09-27. The dq suites fail on main. Eleven cases do, in three unrelated groups; two of the three are fixed here and the third is diagnosed and left for its own unit, because it cascades further than this one should carry.

Seven generator assertions were stale. generators_tests.cpp asserts sut.version = 1= in seven of its twelve *_generator_produces_valid_instance cases and = 0= in the other five, while every generated generator sets version = 0 through the same template line. Nothing distinguishes the two groups: the split runs across catalog, coding_scheme, dataset, methodology, nature_dimension, origin_dimension, subject_area, treatment_dimension and the change-reason pair with no pattern. Zero is the right answer, because the write path reads version zero as "no current row exists", which is exactly what a freshly generated instance is. The seven now say zero. The reason this survived is worth recording: one assertion written twelve times with two different answers has no single place to be right.

The change_reason generator named a category the seed does not create. Its model hard-coded std::string("data_correction"), taken from the model's own prose examples, while dq_change_reasons_populate.sql creates system, common and trade – the three the seed aligns to BCBS 239, FRTB and FINRA. The change-reason insert trigger validates the reference, so a generated reason was refused and write_change_reason_publishes_an_event failed. The model now generates common, and its prose names the three real categories rather than two that never existed.

What is left, and why it is its own unit. The two LEI eventing tests fail because ores_dq_lei_entities_tbl and ores_dq_lei_relationships_tbl do not exist. Both models declare that table name, but dq creates only the artefact staging tables, and both hand-written query-function files read dq_*_artefact_tbl. The generated repository therefore queries a table nothing creates. Correcting the table name is the first of four cascading corrections, and each exposes the next:

1. Point :tablename: at the artefact table.
2. The models declare no :current_state: and no :no_audit_columns:, so the generated entity expects modified_by=/=recorded_at and a temporal pair the staging table does not have.
3. Their ** Table display sections still list modified_by, which the domain table renderer then references.
4. Declaring the entity current-state orphans its history provider registrar and history field mapper – the gate already drops them, so the committed files become stale and stop compiling – and leaves the two eventing integration tests asserting change_reason_code, change_commentary and get_*_history, none of which a current-state entity has.

That is a unit of work, not a line: it removes a temporal surface from two entities and rewrites their integration tests. It is recorded as its own task rather than half-done here.

Evidence. dq goes from eleven failing cases to three – the two LEI cases above and one coding_scheme table-content assertion, which is still to diagnose. The dq api, core, service and shell targets build and link.

Run on 2026-09-27. Replaying the shell's generated commands against a rebuilt database found that ten of them do not answer. The cause is here: dq's report_definition model binds artefact-staging-only, so it is the staging table reporting's entity is imported through – its only SQL is dq_report_definitions_artefact_create.sql – and it nevertheless generated a user-facing command unit. The shell names a menu after the entity's plural, so the shim generated report_definitions, the same menu reporting generates, and dq_commands registers before reporting's unit in repl.cpp. The shim won.

Why nothing noticed. cli::Menu::Insert appends without a duplicate check and dispatch answers from the first menu that has the verb, so the collision is silent and the winner is decided by the order of the calls in repl.cpp. Five commands returned dq's internal error and five were rejected with an arity that belongs to reporting: the recipe is generated correctly for reporting's signature, and dq's menu is what ran.

The gate, and why the profile alone is not the condition. The generator now drops ores.cpp.shell-command and ores.doc.shell-recipe for an artefact-staging-only model whose plural another component also models. The profile alone would have been wrong: three further dq models bind it (lei_entity, lei_relationship, synthetic_fx_spot_config), dq is their only owner, and their command units are the only surface those entities have. Suppressing them would trade a collision for a lost capability. The ownership test reads every component's modeling directory, cached per run, on the same reasoning as the operation-protocol scan.

Evidence. project(ores.dq.report_definition) now resolves no shell unit and no recipe, and the other three resolve both. Regenerating dq's ores.shell and ores.doc facets writes nothing, so the unit is gone rather than re-emitted. The duplicate menu count in the shell falls from five to four; the remainder are ores.shell's and are carried by the shell command collision story. Codegen suite 805 passing, including three new tests that fail when the gate is disabled and only those; the shell and its dq tests build and link; twin coverage, handler permissions, registry exceptions, populate references, the shell component file lists, the recipe inventory, validate_docs.sh and compass lint all pass.

Run on 2026-09-27. ores_dq_dataset_publications_tbl is modelled, so every table-backed entity in dq now has a model. Seven of the twelve hand-written protocol headers are gone: publication_protocol.hpp is generated.

The subject does not move. The generated canonical list subject is dq.v1.publications.list, exactly the subject the hand-written protocol declared, and nothing in the tree sent the old one. So the endpoint keeps its wire identity while its protocol, handler, service, repository, mapper, entity and domain all become generated.

The model's one piece of vocabulary is :current_state:. The log is append-only, so the model declares :current_state: true with :no_audit_columns: true and no natural key. The generated table matches the hand-written one column for column and check for check, with two recorded differences:

• The primary key lost default gen_random_uuid() because the template dropped a declared default on a key column. That defect is now fixed at source and landed on this branch: projects/ores.codegen/library/templates/ores.sql.schema.domain_entity_create.org renders {{#default}} default {{{default}}}{{/default}} in the primary-key block, projects/ores.sql/create/dq/dq_publications_create.sql now reads "id" uuid not null default gen_random_uuid(),, and projects/ores.codegen/tests/test_pk_column_default.py is a mutation-proven regression test.
• Index names come out publications_dataset_id_idx rather than dataset_publications_dataset_id_idx, because the template derives the prefix from the entity rather than the table name. Nothing references index names, so this is the same cosmetic difference part ten accepted for the FSM pair.

The publishing orchestration had to move, and that was the real work. The entity is named publication, so the service facet generates publication_service – which is exactly where the hand-written publishing orchestration lived: dependency resolution, the publishable-dataset reads, and the artefact-type cache. Those moved to a new hand-written publish_service, which publish_handler and dataset_dependency_handler now use. The orchestration's two run-log reads (get_publication_history, get_recent_publications) went with the run log, and the generated handler is their replacement.

Evidence. Eight targets build and link: ores.dq.core.lib, ores.dq.api.tests, ores.dq.core.tests, ores.dq.service.exe, ores.dq.service.tests, ores.shell.dq.tests and ores.shell.exe. validate_schemas.sh reports 0 warnings and 0 hygiene problems on 1534 files; twin coverage, handler permissions, registry exceptions, populate references and the three component_files.cmake lists pass, as do validate_docs.sh and compass lint.

Run on 2026-09-27. dataset is the first of the two entities part eighteen unblocked, and the hand-written layer it needed is gone: dataset_handler.hpp, dataset_protocol.hpp, dataset_service.{hpp,cpp}, dataset_repository.{hpp,cpp}, dataset_mapper.{hpp,cpp} and dataset_entity.{hpp,cpp} are all generated now. Six of the twelve hand-written protocol headers are gone.

What was blocking it was four facet opt-outs, not the model. The model was already written and its table verified in part nine, but it carried :ores.cpp.protocol.enabled: false, :ores.cpp.nats-handler.enabled: false, :ores.cpp.nats-sub-registrar.enabled: false and :ores.cpp.service.enabled: false – the workaround part nine recorded when the entity could not migrate. Removing them is the whole of what the model needed; it then rendered 37 files.

What the generator now writes. The protocol, handler, service, registrar, eventing header, event registrar, history provider registrar, history field mapper, the shell command unit with its test, the TS domain and protocol twins, and the table, drop and notify trigger. The registrar composes register_dataset_handlers and register_dataset_history_provider; the service registers register_dataset_event_mapping. dq_commands lists the new shell unit.

Three things had to change by hand, and two are the same lesson.

• publication_service held a stateful dataset_repository and called read_latest(id). The generated repository is stateless and reads the primary key as a string, so the member lost its constructor argument and the four call sites pass ctx_ and boost::uuids::to_string(id).
• The hand-written domain disagreed with its own table. artefact_type is text not null in both the old and the generated DDL, but the hand-written C++ made it std::optional<std::string>, and publication_service dutifully checked has_value() on a column that cannot be null. The generated domain is std::string, so thirteen call sites dropped their optional handling. The generator was right and the hand-written type was wrong.
• repository_dataset_repository_tests.cpp moved to the generated API, and now makes each generated dataset's code and name unique: the table enforces two natural keys, and the generator draws both from a small word list, so two members of one batch can collide before the behaviour under test is reached.

The singular/plural split in the SQL is closed. dq_dataset_create.sql and dq_dataset_drop.sql are deleted, and dq_create.sql and dq_drop.sql include the generated plural files. That is the shadowed pair part fourteen recorded.

Evidence. ores.dq.core.lib, ores.dq.api.tests, ores.dq.core.tests, ores.dq.service.lib, ores.dq.service.exe, ores.dq.service.tests, ores.shell.dq.tests and ores.shell.exe build and link, as does ores.synthetic.core.lib, which consumes the dq dataset domain. validate_schemas.sh passes with 0 warnings on 1532 files; twin coverage, handler permissions, registry exceptions and populate references pass; the four dq and shell component_files.cmake lists are up to date; validate_docs.sh and compass lint pass.

Run on 2026-09-26. The separation part seventeen named is done. dataset_handler and publication_handler no longer serve a publish verb, so both entities can now migrate.

One handler, two verbs. publish_handler.hpp is new and hand-written. It serves both publication verbs: publish_datasets names the datasets to publish, and publish_bundle names a bundle. Both resolve their datasets to the same bundle_publishable_dataset entries and dispatch the same bundle_publish_workflow, so the class also pays for itself. The two methods shared a tenant lookup, a workflow request, an instance id and a dispatch, and that shared part is now one private start_publish_workflow method rather than two copies.

The verb is a model now, so its protocol is generated. publish_datasets is declared as a second ores.codegen.operation beside publish_bundle, and it renders publish_datasets_protocol.hpp and its TypeScript twin. dataset_protocol.hpp therefore no longer carries publish_datasets_request or its response; the two structs move to the generated header with the same names, subject and fields. mode is a string here for the same reason it is on the bundle verb: the publication mode is a hand-written enum with no model, and the twin gate refuses an opt-out.

What moved where. dataset_publish_params moved out of the anonymous namespace of dataset_handler.hpp into publish_params.hpp, beside publish_bundle_params, with the matching build_params_json overload. The registrar composes one publish_handler instance and subscribes both subjects to it. Nothing else in the tree sends either request, so no caller changed and no SQL changed.

Evidence. The dq core library, the dq service and the shell executable build and link. Twin coverage reports 45 generated headers in 8 components, all twinned; handler permissions, registry exceptions, populate references, cmake component files, shell recipe inventory, physical space inventories, validate_docs.sh and compass lint all pass. The drift check compares the whole tree against HEAD, so it runs after the commit rather than before it.

Next. The two stashed entities. dataset and publication are modelled and their tables were verified in parts nine and thirteen, and the block that stopped both – a handler that owns a publish verb – is now gone.

Run on 2026-09-26. The sequence part sixteen predicted is done, and publish_bundle_protocol.hpp is generated. Five of the twelve hand-written protocol headers are now gone.

The refactor was the whole of it. The params helpers moved to publish_params.hpp and six consumers across dq, iam and shell include it: party_provisioning_plan.hpp, dataset_handler.hpp and publication_handler.hpp in dq, tenant_provisioning_handler.hpp in iam, and provision_commands.cpp and bundles_commands.cpp in shell. The operation model then rendered the request and response to the same path with the same structs, subject and fields, and the handler stayed hand-written beside it. All seven affected targets build and link.

Two things had to be found by building rather than by reading. The generated TypeScript twin imported ../domain/publication_mode.js, a module that does not exist because publication_mode is a hand-written C++ enum with no model. Opting the operation out of the TypeScript facet is not available: the twin-coverage gate refuses any model that sets :ores.ts.protocol.enabled:, and it scans every model rather than only the components under test. So mode is carried as a string, which is what it became on the wire anyway – the handler stringified it on the next line, and the four shell assignments now write "upsert". The enum stays for callers that want it; the protocol simply does not name it.

What this unblocks, and what it does not. The publish verb now has a modelled message and a separated params header, so the two stashed entities are one step closer. What still stops them is that their handlers own the verb: publication_handler and dataset_handler each serve a publish method beside their entity methods, so migrating either entity drops the operation. The next step is therefore to move those two methods into handlers of their own – now straightforward, because the messages are generated and the params have a home.

Evidence. Codegen suite passing; dq drift clean twice in a row and --all clean; seven targets build and link including ores.iam.core.lib and ores.shell.exe; the script tangle produces no change; twin coverage, model drift, handler permissions, populate references, registry exceptions, compass lint and validate_docs.sh all pass; CI green on all ten checks.

Run on 2026-09-26. An operation model for the bundle publish verb was written and its header generated. The result is close to landable and the reason it is not is specific.

The operation model reproduces the wire shape. ores.codegen.operation generates publish_bundle_protocol.hpp – the same path as the hand-written header – so the generated file replaces it in place, with the same struct names, the same nats_subject and the same fields in the same order. It adds one member the canonical protocol carries and the hand-written one did not: static constexpr bool requires_session = true. That is the pattern that retired coding_scheme_protocol.hpp, and it works here too.

What it does not reproduce is the params payload helpers. The hand-written header also declares lei_parties_params, publish_bundle_params and build_params_json, which are not wire messages, and the operation model has no way to express them. They cannot simply go with the header, because they are shared across three components:

publish_bundle_params   dq: party_provisioning_plan.hpp, dataset_handler.hpp, publication_handler.hpp
                        iam: tenant_provisioning_handler.hpp
                        shell: provision_commands.cpp
lei_parties_params      iam: tenant_provisioning_handler.hpp
                        shell: bundles_commands.cpp
build_params_json       dq: party_provisioning_plan.hpp
                        iam: tenant_provisioning_handler.hpp
                        shell: provision_commands.cpp, bundles_commands.cpp

So retiring this one header means giving those three names a home of their own first – a new header in ores.dq.api, with five or six includes updated across dq, iam and shell – before the operation model can land. That is a small, bounded refactor, but it is a refactor, and it is why the highest-leverage item in the list is not a one-step job either.

What this settles. The operation path itself is proven: an operation model renders the right messages to the right path, and the only friction is hand-written content that shares the file. That is the same shape as the entity problem part thirteen found – a handler sharing its class with a bespoke verb – one level down.

Evidence. No source change: the operation model is in git stash ("wip: dq publish_bundle operation model…"), the working tree is clean and check_component_drift.py --component dq reports no drift on main.

Run on 2026-09-26. The smallest item from part fourteen is closed. The coding_scheme model now opts in to the artefact archetype, so dq_coding_schemes_artefact_create.sql is generated rather than hand-written. The generated table matches the hand-written one column for column.

One difference was accepted deliberately. The generated table has "version" integer not null where the hand-written file had default 0. Every other generated artefact table in the component has the same no-default form, so the hand-written file was the deviation rather than the template – and nothing inserts into this table, because the artefact upsert function takes the version as a parameter. The alternative was to change the artefact template, which would have altered every artefact table in every component to preserve one file's local quirk. That is the same trade part ten made for the composite index names: a cosmetic difference is accepted, with the reason, rather than paid for across the tree.

The classified census is now one file shorter: 80 generated, 28 hand-written.

Run on 2026-09-26. create/dq held 108 files then, of which 79 were generated and 29 hand-written; the tree now holds 109 files, 84 generated and 25 hand-written. The acceptance criterion is that each is "generated from a model, or recorded with the reason it is not", so this part classifies all 29 files that were hand-written at the time. Nothing here is changed; it is the reason each file has none yet.

What this buys, and what it does not. The acceptance criterion for hand-written SQL is now satisfied by record for all 29 files. The standard is not: nine protocol headers still have no model, and part thirteen showed four of those are blocked behind one publish-operation model. Three items in the table above are plain work rather than decisions and are the natural next steps, in order: the coding_scheme artefact opt-in (smallest), the bundle_publication run log (same work as the model in hand), and the population functions (a check, then probably a deletion).

Evidence. No source change: the working tree is clean and check_component_drift.py --component dq reports no drift on main. The classification was built from the generated marker on each of the 108 files, from dq_create.sql's includes, and from the presence or absence of a model per entity.

Run on 2026-09-26. The dataset_publications run log was modelled and its table verified: columns identical, and the primary key and all ten check constraints identical.

The vocabulary holds up, including the parts part seven doubted. readonly-timeseries with :current_state: true gives a read-only surface over a non-bitemporal table – no transaction-time pair, no GIST exclusion, no version, and no put or delete verbs – which is exactly what an append-only log wants. And the types and defaults need no new vocabulary at all: the SQL template renders {{type}} and default {{{default}}} verbatim, so :type: bigint gave bigint and :default: gen_random_uuid() and :default: current_timestamp gave the function defaults the hand-written file has. Part seven listed the run logs as needing a vocabulary decision; they need none.

Two small things did need fixing, and both are template-level. The primary-key column line rendered {{type}} not null with no default, so a key with default gen_random_uuid() lost it; both branches of that line now render a declared default. And the model must not set skip_uuid_check, or the nil-uuid check the table wants disappears – a suppression flag that reads as a convenience silently removes a constraint.

Why it cannot land, and why that is the whole tail. publication_handler serves publish_bundle_request as well as list_publications, and publish_bundle is wired into the bundle-publish workflow. dataset_handler does the same with publish_datasets_request. So neither entity can migrate without dropping a live operation, and the same is true of every remaining table-backed entity: their hand-written handlers share their class with a bespoke operation verb.

That reframes the list. It is not five unrelated migration problems:

• dataset and publication are blocked on the dataset.publish and bundles.publish operations.
• publish_bundle_protocol.hpp and publish_from_dq_protocol.hpp are those operations.
• lei_entity_summary_protocol.hpp and report_definition_template_protocol.hpp serve no table, so they are projections rather than entities.
• badge_protocol.hpp is hand-written by design.

So one unit unblocks four of the nine: an operation model for the publish verbs, which the codegen already supports – ores.codegen.operation renders messages with a :subject: and :response: from one source and leaves the handler hand-written beside it, which is exactly the shape these verbs need.

Evidence. No source change: the run-log model and the primary-key default fix are in git stash ("wip: dq publication run-log model…"), the working tree is clean and check_component_drift.py --component dq reports no drift on main. Correction: both have since landed on feature/dq-clean-batch-2, and the primary-key default fix is covered by projects/ores.codegen/tests/test_pk_column_default.py.

Run on 2026-09-26. The FSM migration is merged. fsm_protocol.hpp, fsm_handler.hpp and fsm_service.{hpp,cpp} are deleted, the registrar composes the two generated registrars, both shell units are registered, and both tables regenerate identically to their hand-written originals. Four of the original twelve headers are now gone – dimension_protocol.hpp, data_organization_protocol.hpp, coding_scheme_protocol.hpp and fsm_protocol.hpp.

Two generator defects came out of it, of the same kind. Both the history field mapper and the SQL mapper rendered a uuid column unconditionally as boost::uuids::to_string(v.x), so a nullable uuid natural key produced code that does not compile. Two things had to change, and the second is the interesting one:

• Both templates gained a nullable arm – the history mapper renders v.x ? boost::uuids::to_string(*v.x) : std::string{}, the SQL mapper guards the assignment with if (v.x).
• The loader now sets the render_is_* flags on natural-key and primary-key columns as well as on plain columns. That is why the columns loop already handled an optional uuid while the natural-key loop did not: the natural-key columns had no flag for the branch to test, so a template-only fix would have changed nothing.

Both are pinned by a new test module, mutation-proven: reverting the loader extension fails both tests and only those. Neither defect is specific to dq – any modelled entity with a nullable uuid and a history surface hits them, which is why they were fixed at source.

The recorded mistake did not recur. Part eleven said that anchoring a registrar block replacement on the payload line leaves the closing })); behind, and that the anchor must be the closing line. It happened again in this change, and because it was written down it was recognised and fixed in one step rather than diagnosis. That is the case for recording the shape of a mistake and not just the instance.

What the remaining five are. dataset is modelled and verified and in git stash, waiting on its publish operation, which the hand-written handler owns and the generated stack has no equivalent for. The two publication run logs need function defaults, bigints and a non-bitemporal key. lei_entity_summary_protocol.hpp and report_definition_template_protocol.hpp serve no table at all, so they need a projection archetype or a recorded exception. publish_from_dq_protocol.hpp is an operation with no subjects of its own. badge_protocol.hpp is hand-written by design.

Evidence. Codegen suite 709 passing plus the two new tests; check_component_drift.py --component dq no drift twice in a row and --all no drift; six targets build and link including ores.shell.exe; the script tangle produces no change; every other gate passes; CI green on all ten checks.

Run on 2026-09-26. The FSM migration was carried to the point where it is a complete change apart from one thing, and that thing is a generator defect, not a gap in the plan.

The migration as it stands. The four hand-written files are deleted: fsm_protocol.hpp (the umbrella the per-entity generated protocols replace), fsm_handler.hpp and fsm_service.{hpp,cpp}. The registrar composes register_fsm_state_handlers and register_fsm_transition_handlers in place of its two hand-written subscribes. Both shell units are wired into the dq_commands aggregate, the two component_files.cmake lists are regenerated, and the generated repositories replaced the hand-written ones in place. The two tables were verified identical to their hand-written originals in part ten.

The defect. cpp_history_field_mapper.cpp.mustache renders a uuid column unconditionally as boost::uuids::to_string(v.x). fsm_transition.from_state_id is a nullable uuid, so the generated mapper does not compile:

fsm_transition_history_field_mapper.cpp:39:81: error: no viable conversion from
'const std::optional<boost::uuids::uuid>' to 'const uuid'

The template already handles nullability for strings – the same mapper emits v.guard_function.value_or(std::string{}) for a nullable text column – so the omission is specific to uuids. Two sites need a nullable branch: the natural-key loop at {{#is_uuid}} (line 33) and the column loop at {{#render_is_uuid}} (line 58). The fix is at source in the template, with a regression test, because the defect is not specific to this entity: any modelled entity with a nullable uuid and a history surface hits it.

A recurring mistake worth encoding. Replacing a registrar block by anchoring on the payload line leaves the block's closing })); behind, and the result is a syntax error far from the edit that reads as something else entirely. It happened for subject_area and again for the FSM block, at registrar.cpp in both cases. Two occurrences makes it a property of the edit rather than a slip: the anchor has to be the closing line, not the last payload line. A small script that swaps a block by matching its first and last lines together would remove the class of error, and is a better home for this than another sentence of prose.

Evidence. No source change: the wired migration and the verified models are in git stash ("wip: dq fsm pair WIRED (umbrellas deleted, registrar rewired, shell wired) - blocked on history-field-mapper template…"), the working tree is clean and check_component_drift.py --component dq reports no drift on main.

Run on 2026-09-26. Models for fsm_state and fsm_transition were written and both tables verified positively, not by absence of a complaint: columns identical, constraints and checks identical, and the index set identical including its where clauses.

The three vocabulary pieces part eight named are now confirmed on live entities.

• A composite natural key is several :natural_key: true flags. fsm_state flags machine_id and name; fsm_transition flags machine_id, from_state_id and to_state_id, the middle one nullable, and the generated table keys on all of them.
• An extra index is a ** Indexes row under * SQL, and it produced both fsm_states_machine_idx and fsm_transitions_machine_idx on (tenant_id, machine_id) with the right where clause.
• Extra check constraints are a ** Checks table with an expression column, and it produced check ("is_initial" in (0, 1)) and check ("is_terminal" in (0, 1)) exactly as the hand-written file had them.

The one difference is index names, and it is cosmetic. The template derives a composite index name from its columns, so it emits fsm_states_machine_id_name_uniq_idx where the hand-written file chose fsm_states_machine_name_uniq_idx, and fsm_transitions_machine_id_from_state_id_to_state_id_uniq_idx where it chose fsm_transitions_machine_states_uniq_idx. Nothing references those names: neither drop script mentions an index, and dropping the table drops its indexes. So the rename has no effect, and it is worth saying so rather than treating it as a blocked difference.

What the migration still needs, and why it is not in this change. Five deletions and their rewiring: fsm_protocol.hpp, which is the hand-written umbrella the per-entity generated protocols replace; fsm_handler.hpp; fsm_service.{hpp,cpp}; and the two hand-written repositories, which the generated stateless ones replace, so the two repository_fsm_*_repository_tests.cpp files move to that API as the other entities' tests did. The registrar composes register_fsm_state_handlers and register_fsm_transition_handlers in place of the twelve hand-written subscribes, and both shell units join the dq_commands aggregate and the part's file lists.

One thing stays out of scope. fsm_machine has a table but no C++ layer at all – no repository, service or handler. Modelling it would generate a messaging surface the platform never had, so it is a widening rather than a migration and stays where it is.

Evidence. No source change: the two models and the regenerated SQL are in git stash ("wip: dq fsm pair models (both tables verified…)"), the working tree is clean and check_component_drift.py --component dq reports no drift on main. The comparisons above were made with git show HEAD:<path> against the working path, which is the same path the generator writes for these two tables – checked before comparing, which is the mistake part nine records.

Run on 2026-09-26. A dataset model was written and its table verified. It reproduces the hand-written dq_dataset_create.sql: every column (the five nullable ones rendered as text null rather than text, which is the same thing), the bitemporal primary key, the exclusion constraint and the checks, and both natural keys – (tenant_id, code) from the flagged key, and (tenant_id, name, subject_area_name, domain_name) from the extra-index row. The generated table adds one redundant index on (tenant_id, id), which the profile emits for every uuid-surrogate entity. So part eight's syntax works, and dataset is the first entity whose second natural key is expressed by it.

The migration cannot land, because the hand-written handler owns a verb the generated stack does not have. dataset_handler serves publish_datasets_request through a publish method wired into the bundle-publish workflow – it includes bundle_publish_workflow.hpp and publication_service.hpp and carries a dataset_publish_params payload. The generated handler has list, get, put, delete and history. Migrating the stack would silently drop the publish path.

Scoping the model to its SQL facet is not available either. Disabling ores.cpp.protocol, nats-handler, nats-sub-registrar and service on the model still generates the domain, repository, generator, eventing and presentation facets, so the C++ moves anyway; and a generated repository breaks the hand-written dataset_service, which calls the old API – exactly the failure methodology produced before it was trimmed. Restricting a model to one facet is therefore not a smaller migration; it is the same migration with the pieces left ungenerated.

So the order changes again. dataset waits on modelling its publish operation, which is the operation shape part seven listed and is a larger decision than the entity. The FSM pair is the better next unit: it has no bespoke verbs, it is table-backed, and part eight supplied both pieces of vocabulary it needs – two :natural_key: flags for (machine_id, name) and one extra-index row for machine_id.

A verification error worth recording, because it is the second in two rounds. The first comparison in this round reported the generated table as identical to the hand-written one. It was comparing a file with itself: git show HEAD:<singular path> against the working <singular path>, when the generator writes the plural path. The conclusion happens to survive the corrected comparison, but the check did not test anything. Round twenty-one's error was a truncated grep; this one is a wrong path. Both are "checked the wrong thing" rather than "checked badly", and both were caught only by looking again at the output rather than at the claim.

Evidence. No source change: the model and the regenerated SQL are in git stash ("wip: dq dataset model (table verified incl. both natural keys)"), the working tree is clean and check_component_drift.py --component dq reports no drift on main. The table comparison above is the corrected one, against dq_datasets_create.sql.

Run on 2026-09-26. Parts five, six and seven reported that the remaining entities need model vocabulary the component does not have. Most of that was wrong. The vocabulary exists. I did not find it because I searched the models instead of the templates.

A natural key is one group per model, and its members are flagged. subject_area is the worked example: name and domain_name both carry :primary_key: true, which is how the model states a true compound physical key. dataset_bundle flags code and name each :natural_key: true and gets one index on both together, dataset_bundles_code_name_uniq_idx — not two. So a composite natural key is expressed by flagging every member, and the loader collects them into a single group (org_loader.py:943).

Extra indexes are declared in a table. Under * SQL, an ** Indexes table with the columns name | columns | unique | current_only | where_extra, read at org_loader.py:1452 and rendered by sql_schema_domain_entity_create.mustache as {prefix}_{name}_idx. ores.compute.grid_sample is the worked example.

What that means for the two entities this blocked.

• dataset has two natural keys: (tenant_id, code) and (tenant_id, name, subject_area_name, domain_name). One group per model covers the first by flagging code; the second is an extra unique, current-only index on the three columns. So part seven's "nothing structural, authoring only" was wrong in the other direction: it does need a second key, and the second key is expressible.
• fsm_states and fsm_transitions need a composite key on (machine_id, name), which is two flags, plus an index on machine_id, which is one extra index row. Part five's "no natural key anywhere" was wrong, and part six's "three specific vocabulary gaps" was wrong in the same way: two of the three are ordinary declarations.

What remains genuinely different, and is a real decision.

• Function defaults: gen_random_uuid(), default current_timestamp.
• bigint columns.
• A non-bitemporal primary key on id alone, as the two publication run logs have.
• Two protocols with no table at all: lei_entity_summary_protocol.hpp and report_definition_template_protocol.hpp.
• publish_from_dq_protocol.hpp, which is an operation with no subjects of its own.

Those five are the decisions, not seven. And the badge projection is hand-written by design.

A method note, because this is the lesson. Three rounds went into a vocabulary that was already documented in the templates. The move that found it in one step is to grep the template org files for the mustache variables — {{#domain_entity.indexes}} — rather than the models for a use of them. A shape no model exercises still has a template that names it. That should be the first move next time, before any claim that something is unsupported.

Evidence. No source change: the working tree is clean and check_component_drift.py --component dq reports no drift on main. The syntax above was read from ores.dq.subject_area.org, ores.dq.dataset_bundle.org, ores.compute.grid_sample.org, org_loader.py:943 and :1452, and sql_schema_domain_entity_create.mustache:245.

Run on 2026-09-26. Parts five and six established that the remaining nine protocols do not need one more shape repeated. This part states, per protocol, exactly why it cannot be modelled with today's vocabulary, so the decision that unblocks each is a decision and not more discovery.

What this changes about the plan. Seven of the nine need a decision rather than work, and three of those seven are arguably exceptions the standard should record rather than migrations it should force: the badge projection is hand-written by design, and the two summaries have no table to model. Of the rest, only dataset is pure authoring.

So the honest next step is one entity, not a sweep. dataset first, because it needs nothing new; then a decision on the vocabulary list above — composite keys, reference indexes, integer checks, function defaults, non-bitemporal keys — which is a codegen feature in its own right and should be its own story rather than a side effect of this task. The three likely exceptions should be put to the standard as recorded reasons, which the acceptance criteria already allow ("generated from a model, or recorded with the reason it is not").

Evidence. No source change: the working tree is clean and check_component_drift.py --component dq reports no drift on main. The table shapes quoted above were read from the committed create scripts, and the junction template's unconditional version column from sql_schema_junction_create.mustache:20.

Run on 2026-09-26. Part five claimed the FSM tables have "no natural key anywhere". That was wrong, and the error was mine: I grepped for create unique index and read only the first few lines of the output. The tables do carry a composite natural key.

What they actually have. fsm_states has a unique index on (tenant_id, machine_id, name) and a plain index on (tenant_id, machine_id); fsm_transitions is the same shape. Both tables also carry check ("is_initial" in (0, 1)) style constraints on their integer flags.

The vocabulary was tested rather than guessed. A model with a uuid primary key, plain uuid reference columns, integer columns with :default: 0, and profile: uuid-surrogate-lookup regenerated dq_fsm_states_create.sql to within three differences of the committed hand-written file:

• the hand-written table has check ("is_initial" in (0, 1)) and check ("is_terminal" in (0, 1));
• it has the composite unique index and the machine index above.

Everything else was identical: every column and its nullability, the integer defaults, the bitemporal primary key, the exclusion constraint, the nil-uuid check and the stamps. So plain uuid columns, integer defaults, and a uuid key with no single-column natural key all work already — the vocabulary is not missing a whole shape.

The gap is three specific things, none demonstrated by a dq model today. The vocabulary for a composite natural key — two plain columns marked as one key — for an index on a reference column, and for a check constraint on an integer column. They are the next thing to find, and the places to look are the junction models (which already declare a two-column identity) and the ** Validations table the entity scaffold emits but no dq model fills in.

Nothing was left behind. The model and the scaffold were removed and the two regenerated SQL files restored, so the working tree is clean and dq_fsm_states_create.sql still regenerates to its committed content.

Run on 2026-09-26. No migration this time: the round set out to take the read-only group and found that the plan's grouping was too tidy. The remaining nine do not divide into "read-only" and "operations". They divide by what model vocabulary they need, and seven of them need vocabulary the component has never used.

Two protocols serve no table at all. lei_entity_summary_protocol.hpp (dq.v1.lei-entities.summary) and report_definition_template_protocol.hpp (dq.v1.report-definition-templates.list) have no table of their own. There is no dq_lei_entity_summaries_create.sql and no dq_report_definition_templates_create.sql: the first is computed from dq_lei_entities_artefact through dq_lei_entities_query_functions_create.sql, the second is served from dq_report_definitions_artefact. Modelling an entity generates a table, so there is nothing here to model. These two need either a read-only projection archetype or a recorded exception, and that is a design decision before it is work.

The FSM family is table-backed but has no natural key. dq_fsm_machines_create.sql, dq_fsm_states_create.sql and dq_fsm_transitions_create.sql are hand-written. All three are uuid-surrogate tables whose only unique indexes are on (tenant_id, id) and (tenant_id, id, version): no natural key anywhere. fsm_state and fsm_transition additionally carry uuid reference columns (machine_id, from_state_id, to_state_id) and integer flags with default 0, and fsm_transition's from_state_id is nullable. Every dq model so far either keys on a code or name natural key or is a junction, so neither "uuid primary key with no natural key" nor "uuid reference column in an entity" has a precedent in this component to copy.

There is a second problem in the same family. fsm_machine has a table but no C++ layer at all — no repository, no service, no handler. Modelling it would generate a messaging surface the platform never had, which is a widening rather than a migration. So the family is not three migrations of one shape; it is two migrations of an unprecedented shape plus a decision about the third.

The operations group is five headers and two run logs. dataset_protocol.hpp serves list, save, delete and history plus a bespoke publish; dataset_dependency_protocol.hpp serves by-dataset and resolve; publication_protocol.hpp serves list; publish_bundle_protocol.hpp serves publish; and publish_from_dq_protocol.hpp has no subjects of its own, because the registrar subscribes its handler to literal strings. The publish tables are run logs: dq_bundle_publication_create.sql has id uuid default gen_random_uuid(), bigint counters, a boolean, published_at timestamp default current_timestamp, a nullable completed_at, and a non-bitemporal primary key on id alone. Function defaults, bigints and a non-bitemporal key are all vocabulary the dq models do not use.

Where that leaves the plan. The survey's ordering — read-only group, then operations — assumed one more shape repeated. The honest split of the nine is: one hand-written by design (badge_protocol.hpp, the mapping projection); two with no table to model; one family of three tables with no natural key and no C++ for its third member; and five operation headers over behaviour and run logs. Seven need a vocabulary decision before a model can be written.

The next step, in order. The FSM shape is the best first decision because it is table-backed, self-contained and needs no new messaging surface for the two entities that already have one — and the scaffolder already offers a --shape fk-scoped preset for exactly this form, which no dq model exercises yet. Establishing that vocabulary, then dataset and its dependency, then the two run logs, then deciding the two projections, is the sequence that follows from this survey rather than from the earlier one.

Evidence. No source change this round: the working tree is clean, check_component_drift.py --component dq reports no drift on main, and the three scaffolded FSM models were removed rather than left half-written.

Run on 2026-09-26. coding_scheme_authority_type and coding_scheme are modelled, so another protocol leaves the hand-written list.

A correction first. Part three's pull request said eight hand-written protocols remained and then listed ten; the count was wrong, and the list was right. Counting by hand is what failed. The reliable test is the generated marker, and the figure it gives is stated at the end of this section, from that check rather than from memory.

The pair. coding_scheme_authority_type is a code-keyed lookup and coding_scheme is the same shape plus an authority, a subject area and an optional uri, so both follow the dimension pattern. The authority and subject-area columns stay plain text: the hand-written table never carried a constraint, and declaring a foreign key would refuse rows the platform accepts today. The generated tables match column for column – the authority type exactly, and the scheme modulo the explicit null on uri and the redundant index the profile adds on the primary key.

coding_scheme_protocol.hpp is now generated, so it leaves the hand-written list rather than being deleted: the model writes the canonical protocol to the same path, and the old structs go with the overwrite. That is the third way a header leaves the list, after deletion (dimension_protocol.hpp, data_organization_protocol.hpp) and shrinking (badge_protocol.hpp).

What else the pair retired. The registrar composes register_coding_scheme_authority_type_handlers and register_coding_scheme_handlers; the four hand-written singular SQL files give way to the generated plural forms in dq_create.sql and dq_drop.sql; both repository tests move to the stateless API; and both shell units are wired into the aggregator and the part's file lists.

Where the list stands. Measured immediately after this change, with the command that measures it rather than by eye:

for f in projects/ores.dq/api/include/ores.dq.api/messaging/*_protocol.hpp; do
    [ "$(grep -c AUTO-GENERATED $f)" = 0 ] && basename $f
done

That gives nine hand-written dq protocol headers: badge_protocol.hpp (the mapping projection, hand-written by design), dataset_protocol.hpp, dataset_dependency_protocol.hpp, fsm_protocol.hpp, lei_entity_summary_protocol.hpp, publication_protocol.hpp, publish_bundle_protocol.hpp, publish_from_dq_protocol.hpp and report_definition_template_protocol.hpp. Three of the original twelve are gone: dimension_protocol.hpp and data_organization_protocol.hpp deleted, coding_scheme_protocol.hpp replaced by its generated twin. The next part re-measures rather than trusting this line.

Evidence. Codegen suite 709 passing; dq drift clean twice in a row and --all clean; the six affected targets build and link including ores.shell.exe; the script tangle produces no change; every other gate passes.

Run on 2026-09-26. methodology is modelled and the hand-written layer that served it is gone, so two more headers leave the list.

The model. A uuid-surrogate-lookup: a surrogate id, a name natural key, a description and two optional text columns, logic_reference and implementation_details. The generated table matches the hand-written one except for the redundant unique index the profile adds on the surrogate key; no column, primary key or exclusion constraint moved.

What it retired. data_organization_protocol.hpp and data_organization_handler.hpp are both deleted — once the dimensions left, methodology was all either of them still held. The registrar composes register_methodology_handlers; dataset_service loses the methodology section it duplicated, because the generated methodology_service owns those verbs; and the two hand-written singular SQL files give way to the generated plural forms in dq_create.sql and dq_drop.sql.

A third rule the generated repository imposes. The stateless API, as with the dimensions, and boost::uuids::to_string(...) on lookups that pass a surrogate key: the generated read_latest(ctx, id) takes the id as a string, so both id lookups in the hand-written test needed the conversion.

A lesson worth keeping: a mem-initializer line can carry the constructor body. Twice now a constructor has been edited by deleting a line from its initializer list, and twice the result did not compile — once because the list then began with a comma, and once because the deleted line ended in {}. The second is the dangerous one, because the damage surfaces as a syntax error far from the edit.

Evidence. Codegen suite 706 passing; dq drift clean twice in a row and --all clean; the six affected targets build and link including ores.shell.exe; the script tangle produces no change; model drift, protocol twin coverage, handler permissions, populate references, registry exceptions, both inventories, compass lint and validate_docs.sh all pass.

Run on 2026-09-26. origin_dimension and treatment_dimension are modelled, so the dimension family is done and the hand-written layer it needed is gone.

One shape, three times. The models are the nature model with the names, titles and generator suffixes changed, which is what the survey predicted: three identical code-keyed lookups. The generated tables match the hand-written ones column for column.

What the pair retired.

• data_organization_protocol.hpp loses its Origin and Treatment sections and now holds only methodologies, so it leaves the twelve-header list when methodology is modelled rather than now.
• dimension_protocol.hpp is deleted. It was a one-line re-export with no consumers, so the twin gate's list is one shorter.
• dimension_handler.hpp and dimension_service.{hpp,cpp} are deleted; the registrar was their last consumer.
• Four hand-written singular SQL files, with dq_create.sql and dq_drop.sql taking the generated plural forms.
• The two repository tests, rewritten to the stateless API like nature's.

A generated unit that compiled and reached nothing. The generator emitted a shell command unit for each dimension, but the aggregator that registers them, dq_commands, is hand-written, so nobody added them. nature_dimension_commands.cpp had been inert since part one for the same reason. The gap was invisible until the shell binary was linked, where the three symbols were undefined — and the component library did not hold them either, because ores.shell.dq's component_files.cmake had not been regenerated since the entities were modelled.

What that means for the remaining entities. A newly modelled dq entity generates a shell unit that the build will not compile and the REPL will not register until two hand-written steps happen: regenerate ores.shell.dq's file lists, and add the unit to dq_commands. Every remaining entity in the survey needs both. A check comparing the aggregate against the units on disk would catch the omission, and it is recorded rather than written because it belongs to the shell component and not to this task.

Evidence. Codegen suite 706 passing; check_component_drift.py --component dq reports no drift twice in a row and --all reports none; ores.dq.core.tests, ores.dq.api.tests, ores.dq.service.tests, ores.shell.dq.tests build and link and ores.shell.exe links; the script tangle produces no change; model drift, protocol twin coverage, handler permissions, populate references, registry exceptions, both inventories, compass lint and validate_docs.sh all pass.

Run on 2026-09-26. nature_dimension is modelled, so the first of the twelve hand-written protocols is gone and the survey above has been executed once end to end.

The model is small and the table did not move. projects/ores.dq/modeling/ores.dq.nature_dimension.org is a 167-line code-keyed simple-lookup: a code primary key, a name and a description. The regenerated dq_nature_dimensions_create.sql is structurally identical to the hand-written dq_nature_dimension_create.sql it replaces — same columns, same primary-key tuple, same partial unique indexes, same exclusion constraint. Only the generated banner and the copyright year differ. Modelling a table does not have to change it.

Two rules the templates impose, found by breaking them. A #+begin_src cpp :name generator expression has to yield std::string. The template appends "-" + std::to_string(idx) to the code generator and concatenates the name one, and faker::word::noun() returns std::string_view, so string_view + const char* = does not compile. Every migrated sibling wraps with =std::string(...); the reason is now written down instead of copied.

What the migration touched besides the model.

• data_organization_protocol.hpp: the 50-line Nature section, and its nature_dimension.hpp include.
• dimension_handler.hpp: the 120-line Nature block.
• dimension_service.{hpp,cpp}: 48 lines of declaration and 55 of definition, plus the repository member and its include.
• registrar.cpp: the four hand-written nature subscribes replaced by register_nature_dimension_handlers.
• dq_create.sql and dq_drop.sql: now include the generated plural files, and the two hand-written singular files are deleted.
• repository_nature_dimension_repository_tests.cpp: hand-written, not generated, and it had to follow.

The generated repository is stateless, and every remaining entity's hand-written test will need the same edit. The old repository held a context and took write(v); the generated one is default-constructed and takes write(ctx, v), read_latest(ctx) and so on. The test was rewritten to that API, and its tenant_id assignment dropped .to_string() because the domain member is now the typed utility::uuid::tenant_id rather than a string. The generated domain also gained the change_reason_code member the hand-written type never had, which is a column the table always carried — so the migration removed a divergence rather than introducing one.

Evidence. Codegen suite 706 passing; check_component_drift.py --component dq reports no drift twice in a row; ores.dq.core.tests, ores.dq.api.tests, ores.dq.service.tests and ores.shell.dq.tests build and link; model drift, protocol twin coverage, handler permissions, populate references and registry exceptions all pass.

Run on 2026-09-26. Part six found that dq regenerates cleanly and still cannot join COMPONENTS_UNDER_TEST, because twelve of its protocol headers are hand-written for entities that have no model and so can have no TypeScript twin. This is what those twelve are, grouped by what modelling each needs, because the groups are not the same job.

Group one, plain lookups (six entities, three headers). methodology, nature_dimension, origin_dimension, treatment_dimension, coding_scheme_authority_type and coding_scheme are ordinary tenant-scoped entities with list, save, delete and history. The first five are simple: methodology keys on name with two nullable columns, and the other four key on code with a name and a description. coding_scheme is the one with work in it — it carries an authority-type reference and a composite reference to subject_area, the entity whose history identity part six records as composite, so its model has to say how that key is expressed. Modelling the six retires data_organization_protocol.hpp, dimension_protocol.hpp (a one-line re-export of it) and coding_scheme_protocol.hpp.

Group two, read-only projections (four entities, four headers). fsm_state, fsm_transition, lei_entity_summary and report_definition_template are declarative or frozen views: two serve only a list, one a summary of another entity, one a list of templates. They are the reason the archetype's write verbs cannot simply be switched on for every model, and the read-only profile exists for exactly this shape.

Group three, operations (four headers). dataset_protocol.hpp, dataset_dependency_protocol.hpp, publication_protocol.hpp and publish_bundle_protocol.hpp serve behaviour rather than rows: publishing a dataset, resolving dependencies across them, listing publications, publishing a bundle. publish_from_dq_protocol.hpp is the extreme case, with no subjects at all — the registrar subscribes its handler to literal strings. These are the candidates for the operation model the operation-protocol-ownership gate already knows about, and they are last because they are the least like the entities modelled so far.

What the scaffold does, and what it leaves. compass add entity_org --component dq --slug <entity> --shape simple-text-key writes a skeleton at projects/ores.dq/modeling/ores.dq.<entity>.org: the frontmatter, the table name, the presentation knobs and the section headings. It does not fill the primary key, the natural keys, the columns or the presentation tables — those are the author's, and the skeleton carries a TODO at each. The hand-written create/dq/dq_<entity>_create.sql is the specification for them: it states the columns, the primary-key tuple and the partial unique indexes, so the model has to reproduce that table or the regeneration rewrites it. That is the per-entity work this section exists to size.

Order. The three dimensions first, because they are one shape written three times, so they prove the pattern and retire two headers at once. Then methodology, which shares its header with them and retires the rest of it. Then coding_scheme_authority_type and coding_scheme, which are the same shape plus the composite key decision. Then the read-only group. Then the operations. Adding dq to COMPONENTS_UNDER_TEST is the last step, once every header has a model behind it.

Run on 2026-09-26, on a branch off the merged main. change_reason, change_reason_category and subject_area are migrated, so all fifteen entities are canonical, and the generated dq shell units compile because ores.shell.dq is now a component the build adds.

The three entities and the shell had to move together. The generated shell unit includes the entity's protocol header, so a unit cannot compile until that entity's protocol is canonical; and component_files.cmake lists every file in the part's directory, so a part cannot compile thirteen units and skip a fourteenth. That is why the last protocol migration and the shell wiring are one change rather than two.

Wiring the part. ores.shell.dq is an adapter in the ores.shell.refdata shape: hand-authored CMakeLists.txt files, because an adapter's link libraries belong to the shell surface and not to the domain; two generated component_files.cmake lists; a component_overview.org and a diagram; and one hand-written aggregator, dq_commands, that lists the units for the REPL to call. The thirteen submenus replace the two hand-written change-reason commands, and those four files — the two headers and their two implementations — are deleted.

The composite root is declared, not discovered. Adding the directory to projects/ores.shell was not enough, because the generated root adds one add_subdirectory per part in #+parts:, and CMake never learns an undeclared directory exists: configure succeeds and the part is silently absent from the binary. #+parts: gained dq and the root was regenerated. The generator refuses to render a part its composite does not build, so the mistake surfaces where it is made rather than in a binary that quietly lacks a feature.

subject_area keeps a hand-written history bridge, and that is the decision the task was waiting on. Its history identity is the composite (name, domain_name), which the Qt client sends as name|domain_name. The generated history-provider registrar declines to register a provider for a compound key — it says so in the generated source — so the hand-written provider that splits the string stays, with the reason recorded beside it. The alternative was to change the wire identity a shipped client uses, which is not a codegen decision. Everything else about the entity is generated.

Three deletions the migration required, measured. data_organization_protocol.hpp lost its whole subject-area section: 55 lines, and the subject_area.hpp include with it. data_organization_handler.hpp lost 131 lines — the four subject-area handlers. The bespoke data_organization_service lost 62 lines of declaration and 79 of definition, keeping only what dataset_dependency_handler reads. The umbrella could not be deleted outright because methodologies and the three dimensions are still hand-written; it shrank to what it still owns.

A generator defect, found because the shell facet finally compiled. ores.shell.dq has a unit for badge_mapping that includes badge_mapping_protocol.hpp, and nothing generates that header. badge_mapping is the only model in the tree that sets :ores.cpp.protocol.enabled: false — it suppresses the protocol, handler and service because a hand-written read-only handler serves the badge lookup — and the shell facet ignored the opt-out and emitted a unit that could not compile. It is the same dependency the operation-owner gate already encodes, missing for the model's own opt-out, so the fix is one constant: _PROTOCOL_DEPENDENT_FACETS names the facets that carry the protocol's request types — the shell unit, the eventing facets, and the literate recipe that documents the shell unit's commands — and they are dropped when the protocol is not rendered. Two tests pin it: the live badge_mapping junction resolves none of them and keeps its SQL, and a junction that keeps its protocol keeps its shell unit, so the gate cannot pass by dropping the shell facet everywhere.

What follows the unit it documented. The badge_mappings shell recipe and its eight tangled library scripts are views of a unit that no longer exists, so they are deleted with it and the recipe inventory drops the category. The recipe is in the gate rather than left to a model opt-out because a recipe is where its command unit is: the dependency is resolved, not restated per model.

The drift claim, and the gate dq still cannot join. check_component_drift.py --component dq reports no drift twice in a row, which is the property the whole task was aimed at. That makes dq look ready for COMPONENTS_UNDER_TEST, and it is not: check_protocol_twin_coverage.py demands a TypeScript twin for every protocol header of a listed component, and twelve dq headers are hand-written for entities that have no model — datasets, coding schemes, publications, the FSM family, dimensions, methodologies, report-definition templates, and the badge mapping projection. None of those twelve can have a twin generated, so listing dq would fail that gate on all twelve. The reason is recorded in component_registry.py beside the list, so the next reader meets the blocker rather than rediscovering it. Modelling those entities is step 8's remaining work, and it is what admits dq to the list.

Evidence. Codegen suite 706 passing; check_component_drift.py --all green; check_component_drift.py --component dq green twice in a row; model drift, protocol twin coverage, handler permissions, populate references, registry exceptions, cmake file lists, physical-space inventories and the recipe inventory all pass; ores.dq.core.tests, ores.dq.api.tests, ores.dq.service.tests, ores.shell.dq.tests and ores.shell.application.lib build and link, and ores.shell.exe links with the dq part in it; compass lint and validate_docs.sh clean.

Run on 2026-09-26, on a branch off the merged main. badge_definition, badge_severity and code_domain are migrated: twelve of fifteen entities are canonical.

Nothing was modelled. Part four found that regenerating dq already emits a canonical protocol, handler, registrar, repository, service, event header, history-provider registrar and integration test for all six remaining entities. This unit took the three that no external consumer blocks. The change is the regeneration plus the deletion of what it supersedes, exactly as part four predicted.

Three hand-written files shrank to what the generator does not own.

• badge_protocol.hpp went from 213 lines to 45. It kept badge_mapping, the junction projection that has no generated protocol of its own, and lost the legacy badge_definition, badge_severity and code_domain protocols that the generated headers now define. It no longer includes a domain header.
• badge_handler.hpp went from 439 lines to 97. Twelve of its thirteen methods served subjects the generated handlers now own; only list_mappings remains.
• badge_service went from 232 lines to 63 across its two files, for the same reason. It is now a read-only service over the badge mapping junction.

The larger find is that eleven generated history providers existed and only nine hand-written ones were wired. Part three migrated eight entities and committed a generated register_<entity>_history_provider for each, but registrar.cpp kept registering nine providers by hand. The generated units were therefore dead code, and four entities — lei_entity, lei_relationship, report_definition and synthetic_fx_spot_config — had no history provider at all, because the hand-written list never named them. This unit replaces the hand-written blocks with the generated calls. That makes eight previously inert units live, adds the three badge entities, and wires the four that had none. The three deferred entities keep their hand-written providers.

Evidence. Codegen suite 701 passing; check_component_drift.py --all green; model drift, protocol twin coverage, handler permissions, populate references, registry exceptions, cmake file lists, physical-space inventories and the recipe inventory all pass; ores.dq.core.tests, ores.dq.api.tests and ores.dq.service.tests build and link; compass lint and validate_docs.sh clean.

What re-running the generator showed, and what it did not. Regenerating after the unit moved nothing among the twelve migrated entities, which is the byte-identical claim for them. It did move the three deferred entities, because their canonical output is still in the tree's future. So check_component_drift.py --component dq still reports drift, twice over: the deferred entities' output, and the untracked generated units for them and for the shell facet. The unit that lands the last three entities is the one that can claim dq regenerates cleanly, and dq is not in COMPONENTS_UNDER_TEST until then, which is why no gate is red in the meantime.

Run on 2026-09-26. The branch merged origin/main (120 commits) and the drift gate it brought in was read for what it actually guards. Nothing in the remaining six entities has to be modelled.

The generator already emits all six. Regenerating dq produces a canonical protocol, handler, registrar, repository, service, event header, history-provider registrar and integration test for badge_definition, badge_severity, code_domain, change_reason, change_reason_category and subject_area. The committed *_protocol.hpp of each is a legacy 84- to 86-line file that the generator owns and overwrites. So the remaining work is to delete the hand-written duplicates and commit the regeneration, not to author models. Part two read the same group as blocked on missing models; the regeneration shows the blocker is narrower.

Three deletions, named. The three are mechanical once the regeneration lands:

• badge_protocol.hpp (213 lines) defines the legacy badge_severity, code_domain and badge_definition protocols a second time, and badge_mapping, which is a junction with no generated handler of its own. The file shrinks to the badge_mapping projection and its read-only list subject. badge_handler.hpp loses twelve of its thirteen methods in the same change.
• data_organization_protocol.hpp (292 lines) covers methodologies and the three dimensions, none of them modelled, and subject_area. Part two's finding stands for the first four; only the subject_area block goes.
• The generated *_history_provider_registrar units duplicate the hist_registry.register_history_provider calls that registrar.cpp makes by hand. Both sides must go in one change or a provider registers twice.

subject_area needs a decision first, and it is the only one. It has a composite (name, domain_name) natural key and no surrogate, and the Qt client addresses its history as name|domain_name; registrar.cpp splits that string apart by hand. The generated stack has no equivalent for a composite history identity, so migrating it either changes the wire identity that client uses or keeps a hand-written bridge. Recorded rather than guessed at.

What the drift gate actually guards, and what it does not. COMPONENTS_UNDER_TEST in component_registry.py lists eight components: iam, analytics-cpp, assets-cpp, compute-cpp, ore, telemetry-cpp, workflow-cpp and variability-cpp. dq is not among them, and neither is refdata. That is why a committed dq tree can lag its own generator without any gate noticing, and it is why this task's acceptance says "twice in a row" — the property is not yet enforced. Adding dq to that list is the last step, because it is what turns byte-identical regeneration from a claim into a check.

One consequence of that gap, measured. check_component_drift.py --all fails on this branch until two workflow recipes are refreshed: workflow_instances/workflow_instance.org and workflow_steps/workflow_step.org. Both carry optional columns, so the shell sentinel fix writes - where the committed recipes still say __none__. Six lines across two files, committed with the fix. This is the gate working as intended, and it is the only thing standing between the branch and a green drift job.

Regeneration writes into the tree, and that is not free. The gate materializes the output it compares, so running it leaves untracked generated files behind. For dq those are the six entities' event headers, history-provider registrars and eventing tests, and the 42 shell units under projects/ores.shell/dq. They were reverted rather than committed, because they belong with the deletion work above. A reader who finds them untracked has run the gate, not a broken tree.

Also out of scope, and pre-existing. check_component_drift.py --component refdata reports drift on main independent of this branch — upstream template prose and a defaulted operator== that were never regenerated into refdata. refdata is outside COMPONENTS_UNDER_TEST, so no gate sees it. It is not this task's to fix and is not fixed here.

Figures. Codegen suite 700 passing after the merge, up from 687; check_component_drift.py --all green once the two recipes are refreshed; model drift, protocol twin coverage, handler permissions, populate references and registry exceptions all pass; the recipe inventory regenerates to 222 recipes in 143 categories.

Run on 2026-09-26. dataset_bundle and dataset_bundle_member are migrated; nine of fourteen entities are canonical.

Rewiring the registrar was the point. registrar.cpp registered both entities by hand against type names the migration renames, duplicating the generated registrars it already composes. The hand-written blocks are gone and register_dataset_bundle_handlers and register_dataset_bundle_member_handlers are composed with the rest.

A generator defect surfaced, and it is not dq's. The junction branch of cpp_service.cpp.mustache populated its response with response.{{name_short}}, while org_loader builds the junction protocol's collection member from entity_plural_short — which for a junction is the full name. The two disagree whenever a model declares a shorter :name_short::

The result is a compile error rather than a wrong value, which is why the stale committed copies hid it: refdata's own C++ facet carries 479 files of drift, so its services predate the mismatch and still name the full member.

Fixed at source in ores.cpp.service.service_impl.org with a regression test that renders a junction whose :name_short: differs from its name and asserts the response member is the full name. Reverting the template makes exactly that test fail. The repository's own name_short parameter is a different thing and keeps its meaning.

One hand-written consumer followed, as P05 requires: ores.shell/application/src/app/commands/bundles_commands.cpp called get_dataset_bundles_request and now calls list_dataset_bundles_request. Its response member was already bundles — which is dataset_bundle's name_short — so nothing else moved.

The shell-application consumers are now the whole critical path. Three of them block the remaining five entities: bundles_commands.cpp (migrated here), change_reasons_commands.cpp and change_reason_categories_commands.cpp. Each is a hand-written command that duplicates a unit the facet generates into projects/ores.shell/dq, which is compiled by nothing. So the remaining work is: register ores.shell.dq, wire it into the shell's CMake, delete the hand-written triples, and migrate the four entities. That is one coherent unit and it is next.

Verified by build: ores.dq.core.tests, ores.dq.api.tests, ores.dq.service.tests and ores.shell.application.lib all link. Codegen suite 687 passing; validator 0 warnings; component_files lists up to date.

Run on 2026-09-26. catalog, data_domain, lei_entity, lei_relationship, report_definition and synthetic_fx_spot_config are migrated: 155 files regenerated, 22 created, three component_files lists refreshed. Seven of fourteen entities are canonical.

report_definition is ambiguous by name. ores.dq and ores.reporting both declare one, so compass codegen entity generate report_definition refuses. It is generated by org-roam ID instead — what the CLI's "entity name or org-roam ID prefix" wording is for.

change_reason and change_reason_category are held back, and the reason matters. Both are consumed outside dq by hand-written shell commands (ores.shell/application/src/app/commands/change_reasons_commands.cpp and change_reason_categories_commands.cpp) that use the legacy verbs the migration removes. Those commands duplicate the units the facet generates into projects/ores.shell/dq — which, as part one found, is compiled by nothing. So migrating the two means wiring the generated shell unit into the build and deleting the hand-written pair. That is a coherent piece of work and it is the next one; folding it into this commit would have mixed a shell restructuring into a protocol migration.

The other three external consumers are safe, and checking why was the work. ores.synthetic includes dq's catalog.hpp and data_domain.hpp; ores.compute includes change_reason.hpp. All three include the domain class and not the protocol, and the domain diff is the generated marker, documentation, and a defaulted operator==. The equality operator is the only semantic addition and it cannot break a consumer; it exists because the round-trip test needs it. Both components build after the migration.

Verified by build: ores.dq.core.tests, ores.dq.api.tests, ores.dq.service.tests, ores.shell.application.lib and ores.synthetic.core.lib all link. Codegen suite 686 passing; validator 0 warnings.

Seven entities remain, in three groups: the two held back above; dataset_bundle and dataset_bundle_member, whose registrations are hand-written in registrar.cpp against renamed types; and badge_definition, badge_severity, code_domain and subject_area, whose types the hand-written umbrella headers badge_protocol.hpp and data_organization_protocol.hpp also define.

Run on 2026-09-26, while testing whether step 9's drift is safe to apply. It was not, and this is why.

The defect. _sentinel_value in org_loader.py maps a C++ type to "a well-formed value that addresses no row", so a generated script reaches the service instead of failing in the client's parser. It had no entry for any std::optional<T>, so an optional field fell through to the __none__ fallback. The shell cannot read __none__ as T, so the script aborted at the client and checked nothing — the opposite of what it exists for. The absent token is the shell's own: ores.shell.api/app/command_token.hpp defines absent_token = "-" and from_token maps it to nullopt.

It was live, and it was measured. pricing_model_product_parameters add sent __none__ where its optional UUID member is:

-pricing_model_product_parameters add 00000000-... __none__ __none__ __none__ __none__ system.new_record generated_script
+pricing_model_product_parameters add 00000000-... - __none__ __none__ __none__ __none__ system.new_record generated_script

add, set and put-many each sent one wrong token. The scope was tree-wide rather than one entity: 325 fields across the models use one of the six std::optional<...> forms.

The blast radius is exactly that entity. Tangling every recipe in the tree — 1186 scripts — changed three, all of them pricing_model_product_parameters. No other recipe carries such a field, so nothing else needs refreshing.

The first attempt was wrong, and the tests said so. The obvious fix was to refuse an unmapped type, on the argument that a silent __none__ is worse than a loud failure. That broke 14 tests. Reading them showed why: the project already refuses a command whose field a token cannot fill, through unsupported on the projection and the guard at org_loader.py:3884 ("the renderer emits the command regardless, so an unfillable field fails the build"). A second refusal inside _sentinel_value duplicated a better-placed guard and fired on paths that never build a script. The fix is therefore narrow: handle std::optional<...>, add the three scalar types that were also unmapped (utility::uuid::tenant_id, std::size_t, std::chrono::year_month_day), and leave the fallback permissive.

Why it belongs on this task. G07 forbids refreshing a generated artefact while its generator has a known defect, because a refresh bakes the defect in. Six ores.dq models carry std::optional<std::string> columns, so refreshing step 9's drift before this fix would have written the abort into dq's own shell recipes. The fix and its regression test land here as the prerequisite.

Verification. tests/test_shell_recipes.py gains a case per optional form and one end-to-end at _entity_shell_invocation; the existing test_an_unknown_type_still_gets_a_value keeps the fallback's behaviour pinned. Codegen suite 686 passing; validator 0 warnings; inventories 42 up to date; analytics-cpp reports no drift once its recipe is refreshed, and dq, iam and refdata are unmoved.

Run on 2026-09-26. Step 7 said "either absorb the 21 handcrafted files into sql_populate_function_refdata.mustache, or delete the template and record why". The answer is delete, and the 21 files go with it.

The template generates a retired design. It emits ores_dq_preview_<singular>_population and ores_dq_populate_<plural>, the second writing into ores_refdata_<plural>_tbl. Both halves are gone from the architecture: the DQ publish pattern moved publish ownership into the target services as <service>_publish_<entity>_from_dq_fn, and the DQ-side publish functions that wrote to non-DQ tables were deleted. The artefact-type registry records this: ores_dq_artefact_types_tbl carries target_subject where it once carried populate_function, so nothing dispatches to a function by name any more. The one artefact still using the template's own naming, ores_dq_preview_ip2country_population_fn, was a leftover from when the template was wired.

The preview functions have no caller. For all 21, the only reference in the tree is a drop function if exists statement. publish_from_dq_handler.cpp dispatches exactly six functions by subject, all of them publish functions, none a preview. So the previews are dead code, not an unwired feature.

What was removed, in full: the template and its org source; the legacy get_schema_template_mappings entry that named it; 15 files that held nothing but a preview; the four previews in dq_population_functions_create.sql; the preview in each of the five files that also hold a live publish function; and every stale drop function if exists those left behind. Six files remain, holding exactly the six functions the handler and the DQ service call: ores_dq_datasets_list_publishable_fn, ores_dq_ip2country_publish_fn, and the four DQ-owned publishes for badges, code domains and coding schemes.

A serious defect surfaced while doing this. ores_dq_coding_schemes_publish_fn was defined twice, in dq_population_functions_create.sql and dq_coding_schemes_population_functions_create.sql, and dq_create.sql includes both. The first definition expires every active coding scheme for the target tenant before inserting; the second scopes replace_all to the rows that dataset published, validates the mode, and is security definer. The second include wins, so the live behaviour was always the safe one — but which definition was live depended entirely on line order in an aggregator, and the destructive copy sat one reorder away from production. The duplicate is gone; the surviving definition is unchanged, so behaviour is unchanged.

Two smaller findings recorded, not fixed:

• ores_dq_bundle_preview_fn, in dq_bundle_publication_create.sql, is also uncalled. It is a different function from the ones above — bundle-level rather than per-artefact — and it lives in hand-written SQL that step 8 owns.
• dq_population_functions_create.sql's own usage comment advertised the removed preview step. Corrected, and it now records why the previews went.

Figures. Codegen suite 684 passing; validator 0 warnings; inventories 42 up to date. 47 files changed: 2 template deletions, 1 source edit, 21 create files and 21 drop files, plus the two aggregators.

Run on 2026-09-26. The last five moved, and divergent is now empty.

All four domain-entity bodies come out byte-identical to the hand-written column block. No notify trigger changed this time, so no component drift moved with them.

The junction template needed the same switch, and finding that took two tries. image_tag is a junction, and the junction artefact template did not read * Artefact columns at all. The switch itself was straightforward — the section replaces the body, and the key indexes still come from the junction's left and right names. What the first attempt missed is that a junction is loaded by a third loader: * Artefact columns was parsed for domain entities and for lookup entities, but not by load_org_junction_model. The template rendered the projection and the test failed, rather than the section being silently inert — which is the outcome worth having, and the reason the test was written before the model used it.

Two dead helpers went with them, both confirmed dead by a repo-wide search before removal:

• ores_dq_image_tags_artefact_insert_fn, whose only remaining reference was a drop function if exists.
• ores_dq_countries_artefact_insert_fn, removed in the previous batch the same way.

Two latent findings recorded, not fixed. Both sit in hand-written SQL that step 8 owns, and neither is on a live path:

• ores_dq_image_tags_artefact_upsert_fn inserts (dataset_id, image_id, tag_id) while tenant_id is NOT NULL, so it would fail if anything called it. Nothing does; the census found it write-only and uncalled in step 2.
• accounts.photo_key is written by all four Acme populate scripts and read by nothing. business_unit_code looks the same but is not: iam_publish_from_dq_create.sql documents it as "informational only today", with no business-unit FK on ores_iam_accounts_tbl.

Figures. Census: generated 36 to 41, divergent 5 to 0. The remaining buckets are 1 derivable (coding_schemes, which needs a model) and 7 orphan. Validator 0 warnings; codegen suite 684 passing; iam, assets-cpp, refdata and dq all report the same drift as before, because a hand-written staging table was never in the drift and no notify trigger changed this time — refdata stays at 513 and dq at 507.

Run on 2026-09-26. Six staging tables moved to generated: business_units, books, currencies, currency_pair_conventions and portfolio, each with a declared * Artefact columns section, and calendar with none.

When a section is required, and when it is a choice. An artefact table must carry every column a populate script writes and every column a publish function reads. Its own base table is not the test — that is what made the census read as 12 divergences when some were only staleness.

• Required when the default projection drops or mistypes something the pipeline uses. Five of the six below are this case.
• Optional when the projection only adds columns nothing reads. Adding them is harmless and keeps the artefact in step with its base, so the default is left alone. A section may still pin the exact contract, and countries does: its exclusion of coding_scheme_code is load-bearing documentation, because the publish takes that value from the dataset rather than the row.

That distinction is why calendar takes no section and countries does, even though both are the "the hand-written file was merely stale" case.

The six, and what the pipeline needs:

Two difference classes step 3 did not have. Both surfaced here, and both are cases where the import contract is deliberately not the store contract:

• Type translation. portfolios carries is_virtual as integer in the import and boolean in the store. ores_refdata_publish_portfolios_from_dq_fn converts with (m.is_virtual ! 0)=, so the artefact must stay integer and the populate scripts' 1=/=0 literals keep working.
• Nullability divergence. books permits a null rates_centre_code and the store does not. The publish implements the difference with coalesce(a.rates_centre_code, v_root_party.business_center_code), so making the artefact column NOT NULL would leave that fallback unreachable and a future dataset could no longer say "region-agnostic".

Step 3's six classes stand; these are a seventh and an eighth.

A lever for the transcription. scripts/emit_artefact_section.py turns a hand-written CREATE TABLE into the model section. It prints and never writes, so the section is reviewed before it is inserted. It exists because transcribing by eye loses a =null= silently: the SQL still parses and a nullable column quietly becomes NOT NULL. The script is a transcription aid, not a verdict — a stale hand-written table transcribed faithfully is still stale, which is why each of the four was checked against its populate and publish first.

All six bodies carry the hand-written contract. books, business_units and portfolio come out byte-identical to the hand-written column block. currencies and currency_pair_conventions differ only by an explicit null on columns that were already nullable, and calendar only by the defaults and extra nullable columns described above.

Figures. Census: generated 30 to 36, divergent 11 to 5. Validator 0 warnings; codegen suite 681 passing; refdata drift 516 to 513, exactly the three notify triggers that regenerated, and dq drift unchanged at 507 because a hand-written staging table was never in the drift. Regenerating also applied the pending drift to the books, business_units and portfolios notify triggers, whose event payload becomes canonical — G01 work the same touch required. calendars' trigger was already current, which is why four staging tables added one drift file rather than two.

The five that remain, with their verdict already taken:

The one image_tag entry needs the junction artefact template to read * Artefact columns the way the domain_entity one does; it does not yet.

Run on 2026-09-26, while working step 4. Step 3 classified 12 tables as divergent because the hand-written artefact differs from its base table. That measurement is against the hand-written file, and a hand-written file can differ because nobody regenerated it rather than because the import contract genuinely differs. The census cannot tell the two apart. The authoritative test is to enable the facet and read the diff, and it splits the 12 in two.

countries is landed, and it is the stale-file case. Its hand-written artefact was missing coding_scheme_code and carried a dead ores_dq_countries_artefact_insert_fn. Switching the facet on wanted to add the column back. Adding it would have created a dead column, because ores_refdata_publish_countries_from_dq_fn takes coding_scheme_code from the dataset it publishes rather than from the staged row. So the model declares an * Artefact columns section that lists the staging columns and leaves coding_scheme_code out. That is the feature built in step 11 earning its place: the diff is now image_id uuid becoming image_id uuid null plus the removal of the dead insert function, and not one column changes.

Nothing calls ores_dq_countries_artefact_insert_fn; the only reference left is a drop function if exists in the drop script, which is a no-op once the function is gone. Emptying it is H02 dead-code removal, not a loss.

currencies is the genuine-divergence case, and it is next. Its artefact carries holiday_calendar, which the base table does not have. That column is live: populate/iso/iso_currencies_artefact_populate.sql writes it and populate/refdata/refdata_currency_calendars_seed_populate.sql reads it from the artefact to derive currency-to-calendar assignments. So the projection must be told to keep it, exactly as step 3 said an import-only extra requires.

Consequence for the plan. The remaining divergent count overstates the work. Each of the 11 needs the same five-minute test, and only some will need a section. Step 3's six difference classes stand as the vocabulary for writing one; they are not a prediction that all 12 need one.

Run on 2026-09-26. Step 4's page said "the owning component's model gains the artefact facet" for all six. That holds for the three domain entities and is impossible for the three junctions, because no junction could reach a staging table at all. This part builds the missing route.

Why the domain_entity archetype cannot serve a junction. load_org_junction_model returns a {junction: ...} model whose columns live under left, right and columns. An entity's columns live under natural_keys and columns at the model root. So the entity archetype read fields that do not exist on a junction, and the first attempt to opt a junction in produced nothing rather than an error — the inert assignment M04 warns about. That property was reverted in part one.

What was added.

The regenerated tables are byte-identical in body. For currency_calendars the diff is the generated banner, the section comment, the index names gaining their dq_ prefix, and the table name being quoted — not one column changes. The other two currency junctions are the same. badge_mappings differs only by version integer not null default 0 becoming version integer not null, which is safe because dq_badge_mappings_artefact_populate.sql names version explicitly.

Four tables landed here, not three. badge_mappings was sitting in the derivable bucket for the same reason: it is a junction, so it had no route either. Opting it in closes it.

Figures. Census: generated 25 to 29; foreign 6 to 0; derivable 2 to 1. The one that remains is coding_schemes, which needs a model rather than a flag. Validator 0 warnings; codegen suite 681 passing; refdata drift 516 and dq drift 507, both unchanged, because a hand-written staging table was never in the drift.

Run on 2026-09-26. The foreign bucket held six staging tables whose base table belongs to refdata. Three are now generated; the other three turn out to be a different shape entirely.

The three domain entities are one-liners. Each gained :ores.sql.schema.domain_entity_artefact_create.enabled: true in its property drawer. The generated staging table is a pure projection of the refdata table, as step 3's census already said.

Two of the three carry a natural key, so this step also depended on the template fix from step 5. Nothing structural changed in the swap. The differences are the generated banner, the model's prose, the index names gaining their dq_ prefix and the table name being quoted, period_multiplier integer becoming period_multiplier integer null, and display_order gaining default 0 from the model. The last is a widening, not a break: an insert that omitted the column used to fail and now takes the model's default.

The refdata side did not move. The regeneration rewrote 15 files across the three entities and only the three dq staging tables differ, so refdata's own create, drop and notify-trigger files were already current. Both drift figures confirm it: refdata stays at 516 and dq stays at 507, because a hand-written staging table was never in the drift in the first place.

The three junctions are not one-liners, and the property is inert on them. get_junction_template_mappings maps a junction to exactly one archetype, sql_schema_junction_create.mustache, and no junction artefact archetype exists. So a junction cannot reach the staging path at all today. Their hand-written staging tables are simple — dataset_id, tenant_id, the two key columns, version, with an index on each key — so the shape is a clean projection of a junction too, but it needs an archetype written for it.

The property was therefore reverted on all three. Leaving it in place would have been the M04 failure this task is meant to remove: an assignment that is inert without any error.

Figures. Census: generated 22 to 25, foreign 6 to 3. Validator 0 warnings; codegen suite 676 passing; refdata drift 516 and dq drift 507, both unchanged.

Run on 2026-09-26. Three of the five derivable tables are now generated.

Each needed one line, not a model: :ores.sql.schema.domain_entity_artefact_create.enabled: true in the model's property drawer. Step 3 had already classified all three as pure projections.

A template defect blocked them, and it is now fixed. Both artefact archetypes rendered only the plain columns, never the natural keys, and all three entities have exactly one natural key. Switching the archetype on before the fix dropped the name column — the column that dq_badge_definitions_artefact_populate.sql, dq_badge_severities_artefact_populate.sql and dq_code_domains_artefact_populate.sql all insert. The gap was latent because every model already using the archetype had zero natural keys. Fixed at source with a regression test, per G05.

The swap is safe. The regenerated staging tables carry the same columns in the same order as the hand-written ones. The differences are the generated banner, the model's description prose, the index names gaining their dq_ prefix and the table name being quoted, and version integer not null default 0 becoming version integer not null. The default is safe to drop because every populate script names version explicitly.

The create and notify-trigger files for the three entities also picked up drift they already carried: the version-conflict guard and the canonical event payload. That is G01 work the entity's touch required anyway.

Figures. Census: generated 19 to 22, derivable 5 to 2. Component drift for dq: 513 files to 507, exactly the six create and notify-trigger files. iam still reports no drift and refdata still reports 516, so the shared natural-keys marking changed nothing else.

What remains in step 5. badge_mappings is a junction, not an entity, so it takes the junction pathway. coding_schemes has no model at all and needs one authored.

Run on 2026-09-26. This is the generator work step 3 called for, and it unblocks steps 4, 5, 7 and 8.

What was added. The optional * Artefact columns section, read by both artefact archetypes:

Semantics. Absent, both archetypes render exactly what they rendered before. Present, the section replaces the staging table's body after its dataset_id=/=tenant_id header, in the order written. The first entry is the staging key, and the generated key index is built on it; a section-level :key: property overrides that when the key is not written first. Two shapes are refused rather than rendered wrongly:

• an empty section, because a section that declares nothing is a mistake, not a request for the default;
• a section together with :has_artefact_insert_fn: true, because the generated insert helper is built from the entity's own columns and would insert the wrong shape into a table that declares its own. No current model wants both, so the pair is refused instead of guessed at.

Evidence that the default path is untouched. The change is byte-identical for every existing model, which three independent checks agree on:

• compass codegen entity generate account_type --address ores.sql.schema --diff reports no differences. account_type is a generated DQ artefact table.
• check_component_drift.py --component iam --dry-run still reports no drift at all.
• check_component_drift.py --component dq --dry-run still reports 513 files, unchanged from before the template edit.

Evidence that the new path works. tests/test_artefact_columns.py adds 10 tests: the default projection and its key index; the declared body replacing it; the entity columns the section omits being dropped; the key index following the declared key; :key: overriding the first column; the header indexes surviving; both refusals; and the lookup-entity archetype honouring the section too.

The tests are mutation-proven. Neutering _artefact_columns to return nothing fails exactly the 7 section-dependent tests while the 3 default-path tests still pass, so the tests exercise the new behaviour rather than passing alongside it. The tangle is stable: re-tangling produces no further change to either .mustache, so the template-drift gate's git diff --exit-code holds.

Found on 2026-09-26 while scoping steps 4, 5 and 8. It refines Finding 1 of the investigation and re-scopes three plan steps.

The 13 generated artefact tables with no model are exactly the FPML codelist entities. The evidence is an exact match, not a resemblance:

• scripts/fpml_parser.py declares 15 entities in ENTITY_FILE_PATTERNS: account-types, asset-classes, asset-measures, benchmark-rates, business-centres, business-processes, cashflow-types, entity-classifications, local-jurisdictions, party-relationships, party-roles, person-roles, regulatory-corporate-sectors, reporting-regimes, supervisory-bodies.
• 2 of the 15 have an org model today: account_types in ores.iam.account_type and business_centres in ores.refdata.business_centre.
• The other 13 are precisely the 13 tables that carry the generated marker with no entity in the 295-entity registry.

The pathway that produced them is retired. fpml_parser.py does not render SQL directly. generate_entity_model writes <plural>_entity.json, a JSON entity model, and the schema templates render from that. Codegen no longer discovers JSON models: manifest.py records that "JSON model support was decommissioned (all models migrated to org)". Nothing in the repository regenerates these files.

The output is stale, and the staleness is visible. refdata_asset_classes_create.sql carries the AUTO-GENERATED marker, but its insert trigger still uses the pre-codegen shape: new. in lower case, current_timestamp instead of clock_timestamp(), and the old if new.version ! 0 and new.version != current_version= guard with no version-replace check. Every genuinely generated table has the new guard. A second signal says the same thing: the Feb 2026 commit 157f0b6c3c ("Regenerate SQL files with generation headers") added the marker to 113 files by prepending a header, and its diff shows the marker landing on a file whose body it did not rewrite.

Consequence for the plan. The FPML family is not pass 1 work. Authoring its org models needs the dataset archetypes that do not exist yet, and that work is already claimed:

• Migrate FPML, crypto, flags and ip2country onto the org dataset pipeline (backlog, next bucket) owns the migration, and its step 4 deletes the old generators once the new output is byte-identical.
• Its step 3 depends on Add the dataset archetypes the four-dataset migration needs, which includes "FPML per-entity refdata".

So steps 4, 5 and 8 proceed without the FPML family, and the 13 entities are recorded here as a dependency rather than duplicated as work. account_types and business_centres are already generated from org models and stay in scope.

This also corrects the language of Finding 1. Those tables are not unmodelled; they are modelled by a retired generator. The distinction matters, because it changes the fix from "delete an orphan" to "port a model", and it names the port rather than inventing it.

Run on 2026-09-26. The step was named "the two shadowed pairs". The debt register named 49 files, not 2. projects/ores.sql/utility/validation_ignore.txt already recorded them as WIRE_001 with the instruction that "each class is cleared by the component drift story that wires the files into its aggregator and removes the entries". That story is this task, so step 6 cleared all 49.

What changed:

1. Regenerated dq_catalogs_create.sql and dq_subject_areas_create.sql with compass codegen entity generate <entity> --address ores.sql.schema. Both were stale by one template change, the version-conflict guard in the insert trigger.
2. Rewired create/dq/dq_create.sql and drop/dq/dq_drop.sql to the generated plurals.
3. Deleted the five redundant twins: dq_catalog_create.sql, dq_catalog_drop.sql, dq_subject_area_create.sql, dq_subject_area_drop.sql, dq_dataset_bundle_drop.sql.
4. Wired the 44 remaining drops into drop/dq/dq_drop.sql, population functions before their staging table.
5. Removed all 49 WIRE_001 entries and their section comments.

Nothing was lost by the swap. The twins differ only in ways the generated file wins: the generated create carries the version guard and the current description prose, the generated drops are leaner because a table drop already drops its indexes, and the one index whose name differed (subject_areas_name_uniq_idx versus subject_areas_name_domain_name_uniq_idx) is on identical columns, so the generated name is the one that reflects the compound key.

Evidence. validate_schemas.sh reports 0 warnings, and it reports them with the 49 entries deleted from the ignore file rather than suppressed. That is the proof: an unwired file cannot pass silently, because the check that would name it is live. The artefact census also reports 0 duplicated declarations, down from 2. The component drift fell from 517 files to 513: the four regenerated files (two creates, two notify triggers) no longer drift, and ores.sql/create fell from 21 to 17.

Correction to Finding 3 of the investigation. That finding said the generated drop scripts were unreachable. The files are generated, but neither drop template emits an AUTO-GENERATED marker, so a drop file's provenance cannot be read from its header at all. This is not a dq defect: measured across the whole tree, 0 of 600 files under projects/ores.sql/drop/ carry the marker, so it is a repository-wide convention or omission that spans every drop archetype (sql_schema_domain_entity_drop.mustache, sql_schema_notify_trigger_drop.mustache and the rest). It is cross-cutting, it carries its own blast radius far wider than dq, and it is not part of this task's acceptance.

Run on 2026-09-26. Each of the 12 divergent tables was read against its base table and against the publish function that fills the base from it. refdata_publish_from_dq_create.sql carries the evidence for most of them.

The artefact table is the import contract, not a projection of the store. The projection is right by default and wrong in six specific ways, and every one of the six is deliberate. No divergence is a defect.

The evidence for the two classes that are easy to mistake for defects:

• Natural key. ores_refdata_publish_business_units_from_dq_fn inserts unit_type_id by joining ores_refdata_business_unit_types_tbl on code = m.unit_type_code. The staging table holds the code, and the publish resolves it.
• Dataset-supplied. ores_refdata_publish_currencies_from_dq_fn reads coding_scheme_code from ores_dq_datasets_tbl, not from the artefact table, and inserts that value. So the staging table has no business carrying the column. The same function shows party_id arriving from p_params rather than from the row, which is the target-state redaction.
• Not a latent break. refdata_calendars_tbl.source and .is_editable carry defaults, and base_calendar_code and coding_scheme_code are nullable, so the publish's shorter column list is sound.

Decision: the model declares the artefact projection explicitly.

The clean standard forbids two sources of truth, so the import contract belongs in the entity model that already owns the table. Add a * Artefact columns section, the sibling of the * Artefact indexes section that already exists:

• Absent section means the current projection. The 11 pure-projection tables need no change, and neither does any entity whose staging table mirrors it.
• Present section lists the staging columns in order. Each column carries :maps_to: when the publish translates it, naming the base column it becomes, and :source: of row, dataset, publish or default to record where the value comes from.

The section is what makes the difference reviewable. A reader sees the redaction and the translation in the model rather than only inside a PL/pgSQL body, which is the same objection the publish-pattern plan raised about cross-service writes.

Step 4 and step 5 are unblocked. Both now reduce to declaring the projection for each entity once, rather than 11 separate special cases.

Run on 2026-09-26 with scripts/survey_table_consumers.py --product dq. It scanned 33,010 files and reported every reader, writer, registration, drop and C++ consumer of each of the 49 artefact tables.

No artefact table is dead. Every table is written, read or registered by something in the tree, so step 2 deletes nothing. The three earlier findings said the same thing from the other direction: a generated file with no model is still a live table.

The four tables that depart from the common shape:

• ores_dq_image_tags_artefact_tbl. Only ores_dq_image_tags_artefact_upsert_fn writes it, and nothing calls that function. It is never read, and it is not registered. It is the closest to dead of the family, but the function snapshots ores_assets_image_tags_tbl at dataset load, so it is retired only once nothing loads that dataset. Record the decision in step 3; do not delete it here.
• ores_dq_payment_frequencies_artefact_tbl. Registered and read by refdata.v1.payment-frequencies.publish-from-dq, but no populate script in the repository writes it. Either the data is loaded outside the tree, or the pipeline is incomplete. Record which.
• ores_dq_synthetic_gmm_components_artefact_tbl, ores_dq_synthetic_ir_curve_template_entries_artefact_tbl and ores_dq_tags_artefact_tbl. Written and read, but absent from dq_artefact_types_populate.sql, so the bundle publisher cannot dispatch them on their own. The synthetic pair are published through the config that owns them; tags is published through sql_tag_populate.mustache. Confirm that is intended, then either register or record the reason.

The commit to keep, model or retire each table is step 3, which now starts from a clean sheet: nothing is dead, so every table has a model to gain rather than a deletion to earn.

Analysis, 2026-09-26. The full record is the investigation. The figures this task works from:

• create/dq declares 74 tables, 49 of them artefact tables: 19 generated, 5 derivable, 12 divergent, 6 foreign, 7 orphan.
• 13 of the 19 generated artefact tables have no entity in the 295-entity registry. asset_classes is live, so these are not dead by default.
• 84 handcrafted SQL files: 30 artefact, 20 population function, 10 notify trigger, 24 other.
• 21 handcrafted *_population_functions_create.sql files. The template for them is wired to no facet.
• 2 tables declared twice; in both pairs the generated file is unreachable from the aggregators.
• Drift is 517 files: 410 change, 107 create. ores.dq/core 236, ores.dq/api 174, ores.shell/dq 42, ores.dq/service 28, ores.sql/create 21.
• The C++ is generated, not handcrafted: 575 of 651 non-test files are generatable, but only 1 carries the marker.

Levers. Two read-only scripts produce the evidence, and both have a regression test.

• scripts/survey_sql_artefacts.py answers what shape the artefact family has. It pairs a table with its base by entity name across every SQL product. Test: tests/test_survey_sql_artefacts.py.
• scripts/survey_table_consumers.py answers who touches each table. It scans the tree once and records the relation each file has to each name, reading the relation from the statement rather than from the directory. Test: tests/test_survey_table_consumers.py.

Both tests lock a defect that produced a wrong answer first. The census paired tags with image_tags under a suffix rule; the consumer scan missed join reads and the short names in the artefact-type registry. The codegen suite is green at 662 tests.

Open question. The population-function template names its output _population_functions.sql, while the 21 handcrafted files are _population_functions_create.sql. Settle the intended name in step 7.

Trunk condition, not fixed here. regenerate_backlog_indexes.py --check reports doc/agile/product_backlog/inbox.org stale on an unmodified main: six captures exist on disk and are absent from the index. No CI job runs that check. The refresh is unrelated to this task and was left out of the branch.

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Pass 1 is not finished, so this task is not done. What is done is the part the story was built on: artefact tables generate from the same entity model as their base table, the projection is declared in the model, and the handcrafted SQL that no model owned is either gone or has a recorded reason for staying.

The design question is answered affirmatively and the answer is enforced by templates rather than by convention. The census now reads 42 generated, 0 derivable, 0 divergent, 0 foreign and 7 orphan across 49 artefact tables, and none of the 49 is a deletion candidate — so the component shrank in handcrafted SQL without losing a table.

The remaining work is one item: the seven hand-written *_artefact_create.sql files under projects/ores.sql/create/dq that carry no generated marker and declare a table no model owns. Each needs a model, or a recorded reason it has none:

• dq_crm_topology_bundles_artefact_create.sql
• dq_ip2country_artefact_create.sql
• dq_lei_bic_artefact_create.sql
• dq_market_data_observations_artefact_create.sql
• dq_synthetic_gmm_components_artefact_create.sql
• dq_synthetic_ir_curve_configs_artefact_create.sql
• dq_synthetic_ir_curve_template_entries_artefact_create.sql

The three synthetic tables have a partial reason — their owner is not this component. The other four have none. These seven are the reason acceptance criteria 1 and 2 are unmet. Every other item this section once named as remaining has landed: the entities it called "six entities" are migrated, the shell unit is wired, the subject_area composite history identity kept its recorded hand-written bridge, and dq is in COMPONENTS_UNDER_TEST.

Landed as https://github.com/OreStudio/OreStudio/pull/2189, https://github.com/OreStudio/OreStudio/pull/2190, https://github.com/OreStudio/OreStudio/pull/2193 https://github.com/OreStudio/OreStudio/pull/2196 and https://github.com/OreStudio/OreStudio/pull/2198 and https://github.com/OreStudio/OreStudio/pull/2200 and https://github.com/OreStudio/OreStudio/pull/2203.