Story: Clean ores.dq to the component clean standard

Table of Contents

This page documents a story in Sprint 26. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

1. Goal

ores.dq meets the Component Clean Standard: every item: All, Protocol and Entity pass, each item that does not apply is recorded with its reason, and ores.dq joins COMPONENTS_UNDER_TEST. Order 30 of 35 in the sprint 26 component clean-up, tier 3 (domain component).

The component needs two passes. The first gives a model to the SQL that has none, because the standard reads a tree that regenerates cleanly. The second runs the checklist over that tree. The analysis behind the split is the investigation.

2. Status

Field Value
State STARTED
Parent sprint Sprint 26
Now Pass 1 is nearly done: every dq entity is on the canonical protocol stack, the shell adapter is wired, and the component regenerates with no drift. What is left is the entities that still have no model.
Waiting on Nothing. Pass 1 continues.
Next Finish pass 1: the last six entities, the shell wiring and COMPONENTS_UNDER_TEST. See the pass 1 task.
Last touched 2026-09-26

3. Acceptance

  • Every checklist item that applies to a component of kind Entity passes, with its evidence recorded on the task.
  • Every item that does not apply is recorded as not applicable, with the reason.
  • Every generator defect found is fixed at its source, with a regression test.
  • ores.dq is listed in COMPONENTS_UNDER_TEST, and check_component_drift.py --all passes.

4. Tasks

Pass 1 models the handcrafted SQL. Pass 2 runs the standard. Pass 1 is first, because pass 2 verifies a tree that regenerates.

Task Pass State Start End Description
Model the handcrafted ores.dq code and generate the artefact tables 1 STARTED 2026-09-26   Author models for every handcrafted ores.dq table and function, automate the artefact tables, and reach byte-identical regeneration.
Bring ores.dq to the clean standard 2 BACKLOG     Work ores.dq through the Component Clean Standard in order and record each item.

5. Notes

  • Survey on 2026-09-23: 16 entity, junction or operation models; 614 C++ files outside tests, 1 of them generated (0%); 84 messaging headers, 84 of them hand-written.
  • Bound to profiles in sprint 25, but not regenerated since the generated marker landed (#2087), so almost no file carries the marker (G03).
  • Analysis on 2026-09-26, recorded in the investigation: create/dq declares 74 tables, 49 of them artefact tables; 84 handcrafted SQL files; drift is 517 files.
  • The census ran to ground on 2026-09-26 and is now 41 generated, 1 derivable, 0 divergent, 7 orphan — 49 artefact tables, all of them live, so no table is a deletion candidate. Divergent went to zero because the projection is declared in the model rather than reconciled by hand. Progress is on the pass 1 task.
  • The design question is answered: an artefact table generates from the same entity model as its base table, and 19 of the 49 already do. The archetypes drop the audit and bitemporal columns and add dataset_id, and codegen forces the dq_ output prefix for an entity any component owns.
  • The largest automation gain is the publication functions. 21 files implement them by hand, and their template is wired to no facet.
  • 13 of the 19 generated artefact tables have no entity in the 295-entity registry. The drift check cannot see them. asset_classes is one, and it is live.

6. Decisions

  • Split the work into two passes. Pass 1 models the handcrafted SQL; pass 2 runs the standard. Rationale: the standard's generation items (G01 to G03) cannot pass while 84 handcrafted SQL files have no model, and running the protocol and shell items first would touch code that pass 1 replaces.
  • Pair an artefact table with its base by entity name, not by prefix. DQ stages entities other components own, so ores_dq_currencies_artefact_tbl pairs with ores_refdata_currencies_tbl. A prefix rule reports false orphans.
  • Do not treat the generated marker as proof of a live model. 13 generated tables have no model. Delete a table only after a consumer census, never on the marker.

7. Out of scope

  • The publish-path decoupling. The DQ publish pattern plan moved publish ownership to the target services. This story keeps publish-from-dq working; it does not revisit where the functions live.

Emacs 29.3 (Org mode 9.6.15)