Task: Bring ores.dq to the clean standard
Table of Contents
This page documents a task in the Clean ores.dq to the component clean standard story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
ores.dq passes the Component Clean Standard for a component of kind Entity.
This is pass 2. Pass 1, Model the handcrafted ores.dq code and generate the artefact tables, gives a model to the 84 handcrafted SQL files and reaches byte-identical regeneration. Run pass 2 over the tree pass 1 leaves behind.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Clean ores.dq to the component clean standard |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-09-28 |
3. Acceptance
- Every checklist item that applies to a component of kind Entity passes, with its evidence recorded on the task.
- Every item that does not apply is recorded as not applicable, with the reason.
- Every generator defect found is fixed at its source, with a regression test.
- ores.dq is listed in
COMPONENTS_UNDER_TEST, andcheck_component_drift.py --allpasses.
4. Plan
Work the Component Clean Standard in its section order, one entity at a time, and record each item's result in the table below.
- Baseline and inventory (B01 to B05). Record the baseline before any edit.
- Models (M01 to M10).
- Protocol (P01 to P06).
- Generation (G01 to G06).
- Wiring and data (W01 to W04).
- Shell (S01, S02).
- Structure and hygiene (H01 to H03).
- Verification (V01 to V06), ending with the registry entry.
| Item | Result | Evidence |
|---|---|---|
| B01 | Pass. The baseline is the drift check's own output at the branch base, re-measured after every model and template edit. | check_component_drift.py --component dq — "No drift" at each of the six measurement points |
| B02 | Pass. Every model in projects/ores.dq/modeling/ classified by metatype: 22 entity, 2 junction, 3 operation, 1 component model, 2 knowledge documents. Only entity and junction carry variability. |
ls projects/ores.dq/modeling/; check_model_drift.py --summary — "181 of 332 models bind to a profile; 14 profiles defined" |
| B03 | Not assessed. The hand-written C++ survey is pass 1's artefact census. | Pass 1 task |
| B04 | Pass. 210 registered subjects, read from the generated nats_subject constants with a multiline-aware pattern (a single-line pattern misses five), plus the six *.publish-from-dq triggers subscribed as literals. The twenty bare *_events strings are prefixes, not addressable subjects. |
grep -rhozE 'nats_subject[[:space:]]*=[[:space:]]*"dq\.v1\.[a-z0-9._-]+"' projects/ores.dq/api/include; projects/ores.dq/core/modeling/ores.dq.protocol.org |
| B05 | Pass. The two open dq captures were read; each described a defect this batch fixes, and both are retired. No dq branch is in flight in another environment. | doc/agile/product_backlog/inbox/; the two deleted captures |
| B06 | Partial, with a second finding measured. The five constants in the two change_reason* headers that name a code the seed does not provide are deleted. The domain_type_table_io output is also classified: of 26 dq *_table_io.hpp headers, 23 are included by no translation unit but their own .cpp, so the operator<< they define is unreachable by overload resolution and the generic JSON overload in ores.utility/streaming/std_vector.hpp is what callers get; tree-wide 183 of 221 are unreachable, so the decision is cross-cutting and is recorded as a capture rather than fixed here. The remaining public-symbol census is not run. |
projects/ores.dq/api/include/ores.dq.api/domain/change_reason_codes.hpp; change_reason_constants.hpp; doc/agile/product_backlog/inbox/the_domain_type_table_io_facet_renders_an_operator.org |
| M01 | Pass. No legacy JSON model and no lookup_entity remains; every model is on the current org format. |
check_model_drift.py |
| M02 | Fix, with the bind proven rather than assumed. badge_severity and subject_area bound no profile at all, which is what withheld their shell units, recipes and eventing tests. Both now bind simple-lookup. The bind creates ten files and changes none, which is the evidence that the profile matches their actual feature values; a bind that moved existing output would be the forced bind the item forbids. |
projects/ores.dq/modeling/ores.dq.{badge_severity,subject_area}.org; check_component_drift.py --component dq --dry-run — "0 file(s) would change" before the bind |
| M03 | Fix. badge_severity stated :has_tenant_id: true, which simple-lookup already fixes to true. Removed with the bind; regeneration byte-identical. |
projects/ores.dq/modeling/ores.dq.badge_severity.org |
| M04 | Pass. Neither model states a feature outside the drawer the loader reads: their features come from the profile, and the only stated keys are schema, product and component. |
The two models' * Flags drawers |
| M05 | Pass. Every column maps one-to-one to its generated table; no joined or denormalised shape is modelled as an entity. | projects/ores.sql/create/dq/dq_{badge_severities,subject_areas}_create.sql |
| M06 | Pass. Every wire type is modelled and twinned. | check_protocol_twin_coverage.py — "all 77 generated protocol header(s) in 10 committed component(s) have a TypeScript twin, and no model opts out" |
| M07 | Pass. Both junctions name their singular and give each key column its real type; the tenant-scoped RLS is generated. | projects/ores.sql/create/dq/dq_rls_policies_create.sql |
| M08 | Fixed. is_initial and is_terminal in ores.dq.fsm_state were :type: integer with an in (0, 1) check, so a row read as text could be compared with an integer by accident; both are boolean now. The change crossed dq — model, entity, domain, table I/O, protocol and wire schema — the two hand-written ores.reporting predicates that compared them to integers, and five populate scripts whose rows wrote 0 and 1. A one-shot migration drops the two check constraints and converts the columns for a database that predates the change. The item's own task, Convert the remaining integer flags to booleans, names a third column, and it is fixed here too: the dq portfolio artefact's is_virtual was integer against a boolean store and carried a conversion in ores_refdata_publish_portfolios_from_dq_fn, so the staging column is boolean, the conversion is gone, the seeder writes true=/=false, and the four trade test scripts that inserted 0 into the refdata boolean column are corrected. |
projects/ores.dq/modeling/ores.dq.fsm_state.org; projects/ores.refdata/modeling/ores.refdata.portfolio.org; the two files in projects/ores.sql/migration/; check_component_drift.py --sweep — "no file this branch touched would change"; the live shell round trip recorded below |
| M09 | Fix for dq, finding for the tree. Eleven properties in the ** Presentation drawer name retired message types or headers that nothing reads: :save_request_class: dq::messaging::save_catalog_request and its get_*, delete_* and *_message_type siblings, plus :domain_include: and :protocol_include:. Nothing consumes them: no template renders a request_class variable, the loader's feature map has no entry for them, and the only other mentions are in a knowledge page and an archetype that resolve_targets never renders. Removing all 213 of them from 21 dq models leaves every generated file byte-identical, which is the proof they were dead. Tree-wide 75 models in seven other components still carry 321 of these lines, and the skeleton template would hand them to any new model; that is filed as a capture. |
projects/ores.dq/modeling/ores.dq.*.org; check_component_drift.py --component dq --dry-run — "No drift" after the deletion; the capture |
| M10 | Pass, with the root read as a group index. The three parts (api, core, service) each carry all seven sections, their .puml source and a rendered image. The root component_overview.org is a composite index and carries Sub-components and Entity modules instead of Inputs=/=Outputs=/=Entry points; validate_docs.py exempts a grouped composite's root for exactly this reason and never checks it. |
projects/ores.dq/{,api/,core/,service/}modeling/component_overview.org; projects/ores.codegen/validate_docs.py |
| P01 | Not assessed. | — |
| P02 | Fix. Six dq.v1.*.publish-from-dq subjects were written down three times: as literals in the registrar, as keys of a map in the handler, and as a table in the handler's own doc comment. The handler now derives the SQL function name from the subject, as the iam, refdata, assets, marketdata and synthetic handlers already do, and checks the result against an allow-list before it reaches the raw SQL it is interpolated into. The registrar's six near-identical lambdas collapse to one loop. |
projects/ores.dq/core/src/messaging/publish_from_dq_handler.cpp; registrar.cpp |
| P03 | Fix. Two defects of the same shape. dq.v1.lei-entities.summary had no responder, so the shell's lei countries and lei entities failed on every call while the two SQL functions that answer them sat unreferenced; the handler is restored beside the operation model, as report_definition_template_handler already is. And the messaging reference named forty-eight subjects that no longer exist, missing fourteen entities entirely; it now matches the registered surface with zero subjects that the code does not have. |
projects/ores.dq/core/include/ores.dq.core/messaging/lei_entity_summary_handler.hpp; projects/ores.dq/core/modeling/ores.dq.protocol.org |
| P04 | Not assessed. | — |
| P05 | Not assessed. | — |
| P06 | Not assessed. | — |
| G01 | Pass 1. | Pass 1 task |
| G02 | Pass 1. | Pass 1 task |
| G03 | Pass 1, with one defect fixed here. The create archetype's primary-key block dropped a declared default in silence; the only table in the tree whose key declares one is dq_publications_create.sql, which rendered id uuid not null and now renders default gen_random_uuid(). |
projects/ores.codegen/library/templates/ores.sql.schema.domain_entity_create.org; projects/ores.sql/create/dq/dq_publications_create.sql |
| G04 | Not assessed. | — |
| G05 | Fix. Two generator defects fixed at source, each with a regression test in the codegen suite: the primary-key default above, and the protocol-dependent facet gate that asked the run's address instead of the model (see S02). Both tests are mutation-proven — each fails with the fix reverted. | projects/ores.codegen/tests/test_pk_column_default.py; test_protocol_dependent_facets_gate.py; codegen suite 924 passed |
| G06 | Not assessed. | — |
| G07 | Not assessed. | — |
| G08 | Done. The api module had no namespace header although core and service had one; the contract the domain types, table/JSON I/O and protocol schemas share is now stated. |
projects/ores.dq/api/include/ores.dq.api/ores.dq.api.hpp |
| W01 | Done in #2286. | #2286 |
| W02 | Fix. Sixteen permission codes the dq handlers check were seeded nowhere, so no role short of dq::* could be granted them and the UI listed none. The eight families that had no rows at all — artefact types, badge definitions, badge mappings, badge severities, code domains, FSM states, FSM transitions and publications — are now seeded as complete read/write/delete triples, matching the file's own convention. dq also joins check_handler_permissions.py's registry, so the next gap fails a check instead of sitting outside it. |
projects/ores.sql/populate/iam/iam_permissions_populate.sql; check_handler_permissions.py — "7 component(s), 226 distinct code(s)" |
| W03 | Not assessed. | — |
| W04 | Not assessed. | — |
| S01 | Fix. badge_severity and subject_area bound no profile at all, which withheld their units, tests and recipes; both bind simple-lookup now. publication is bound to readonly-timeseries, which enables no shell facet, but its surface is a full list=/=get=/=get_many=/=put=/=put_many=/=delete=/=delete_many and so it needs one; its model states that single override, as ores.iam.session does. All three have generated units, generated tests and recipes. |
projects/ores.shell/dq/; dq_commands.cpp; check_shell_menu_collisions.py |
| S02 | Fix at the root cause. Ten command groups had a unit but no recipe, and the reason no run had ever produced one is a codegen gate: the check that drops protocol-dependent facets asked whether the run's address included ores.cpp.protocol rather than whether the model enables it, and a recipe lives in the ores.doc space, so every doc-only run emptied itself. The gate now asks the model. The ten recipes are regenerated with their library scripts and the inventory, and the tree-wide run the fix enables also refreshed two ores.reporting recipes whose optional-column sentinel had drifted to - unseen, so the gate is green across the registry rather than only for dq. |
projects/ores.codegen/src/codegen/generate.py; doc/recipes/shell/; check_component_drift.py --all --dry-run — "0 file(s)" |
| H01 | Pass. Both passes were run for the first time and the audit record is written in its own section below. The automated pass is build/scripts/generate_component_puml.py, which had never been run against dq: it takes api from 208 boxes to 641, core 88 to 140 and service 4 to 7, and stops the api diagram drawing save_catalog_request and the other names the protocol migration retired. The manual pass was empty in all three and now carries the relationships the automated pass cannot see: 183 protocol-to-domain associations in api, 56 service-to-repository in core, 2 in service, using *-- for a data member and ..> for a signature-only type. All three render clean and the images were read at full size. Recorded as a finding rather than fixed: nothing runs the generator, so running it tree-wide changes 70 of 95 component diagrams and every earlier clean pass accepted a stale or absent one. |
build/scripts/generate_component_puml.py; projects/ores.dq/{api,core,service}/modeling/*.puml; the rendered images read; * Component Architecture Audit record below; the generator capture |
| H02 | Partial. The five constants that name a retired taxonomy are deleted and no commented-out code remains anywhere in the component (a sweep of all six source languages finds none). The dead-code census as a whole is pass 1's. | change_reason_codes.hpp; change_reason_constants.hpp; the sweep in H03 |
| H03 | Fix. The component's only end-of-line comment (publish_service.cpp, inside a boost::adjacency_list template argument list) moves above the code, and the seven comments that narrated the migration history are restated as statements about the code as it is. A re-sweep finds zero remaining. |
projects/ores.dq/core/src/service/publish_service.cpp; projects/ores.dq/core/src/messaging/registrar.cpp; projects/ores.dq/service/tests/config_parser_tests.cpp |
| H04 | Not assessed. | — |
| H05 | Fix, with one part recorded. ores.dq.core's overview named soci as its SQL ORM; the component has zero references to soci and the real dependency is sqlgen. The root index is recorded under M10. |
projects/ores.dq/core/modeling/component_overview.org |
| V01 | Pass. Every affected target builds with zero errors: ores.dq.api.tests, ores.dq.core.tests, ores.dq.service.tests, ores.shell.dq.tests and ores.shell.application.tests. |
The build log reported by compass build |
| V02 | Pass. All five suites pass on the committed tree: ores.dq.api.tests, ores.dq.core.tests (111 cases), ores.dq.service.tests, ores.shell.dq.tests and ores.shell.application.tests. ores.dq.core.tests failed once first, in the eventing integration test the subject_area bind had just created, because the model's synthetic generator wrote domain_name = "reference_data" where the validator accepts the seeded Reference Data. That is a latent defect the new test exposed rather than caused; the generator value is corrected and the suite re-run green. |
compass test run; projects/ores.dq/api/src/generators/subject_area_generator.cpp |
| V03 | Pass. compass db recreate -y -k recreated the schema from scratch and reseeded it, and compass services start brought the fleet up: services: running=25 starting=0 stopped=0 missing=0. Two environment faults had to be cleared first, and neither was the component's. A manual nats-server left over from an earlier session held port 21805, so the fleet's own nats unit failed to bind; and the checkout's .env was stale, which compass env configure regenerated, dropping a dead ORES_HTTP_SERVER_JWT_SECRET that nothing reads and fixing ores.http.server, the one unit that had been dying at startup. |
compass db recreate -y -k; compass services start; compass services status; the capture about the harness |
| V04 | Pass. check_shell_recipes.py replayed every dq recipe against the live fleet: 229 recipes, 181 answered, 48 recorded as not replayed because they are destructive, and no failure. Getting there meant fixing two things the replay exposed. The harness invokes ores.shell directly and needs the checkout's .env exported, which its docstring does not say, and without it every recipe aborted at the sign-in line with a message about login rather than about the connection. And 24 of the library scripts were orphans: ten hand-written sub-recipes whose commands name the retired hyphenated menus, and fourteen for lei_entities and lei_relationships verbs that a read-only entity does not have. The ten documents and all 24 scripts are deleted. |
.audit/clean-shell-recipes/recipes.tsv; the capture about the harness |
| V05 | Pass. Every gate is green with dq registered: component drift (dq and the whole registry), model drift, protocol twin coverage, handler permissions, populate references, shell menu collisions, route auth declarations, registry exceptions, test-case reachability, paste-block symbols, oresmd coverage, and both recipe inventories. The shell script drift gate is also green, and it is the one that caught the missing re-tangle of the two ores.reporting recipe libraries: refreshing a recipe without re-tangling its scripts leaves the script stale even though both were generated. |
The local gate runs in this batch; PR #2291's fifteen checks, twice |
| V06 | Pass. dq has been listed in COMPONENTS_UNDER_TEST since #2286 and every gate runs against it, which is why the item is not the decision it once was: the fleet came up here after the two environment faults above were cleared, so V03 and V04 were verified rather than excepted. Every item in the table now passes; nothing is excepted, so the entry stays and no exception is needed. |
component_registry.py; the item table above |
| V07 | Fix, and proven the way the item asks. A census of all 551 dq test cases found 71 that asserted only a non-throw, a container length, a truthiness or one run against another, in four uniform families: write_single_X and write_multiple_X (only CHECK_NOTHROW), read_latest_X (only emptiness and length), read_all_X_versions (only a length), read_nonexistent_X (size() = 0=, which also passes when the lookup ignores the key), plus the generator and parser cases. All 71 now assert a value the test itself set: the write cases read their row back by its natural key and compare two fields, the collection cases assert each written row is present by key with a matching field, the version cases assert the written descriptions are present, the nonexistent cases write a real row first so a key-ignoring lookup fails, and the generator cases assert the two literals their *_produces_valid_instance sibling pins, for every instance. Re-run census: 0 of 551. Proven by stubbing: with catalog_repository::write made a no-op on both overloads, all seven catalog repository cases fail (write_single, write_multiple, read_latest, read_latest_by_name, read_all_versions, read_nonexistent, roundtrip); before the fix the two write cases passed under that stub. |
projects/ores.codegen/scripts/survey_vacuous_tests.py --component dq; compass test run -R "ores.dq" — three suites, all pass; the stub run recorded on this task |
| V08 | Pass, with every untouched file given its reason. Static include reachability over 331 source files, seeded from 79 test sources and following each reachable header's implementation, puts 154 behind a test and 177 not. The 177, grouped with the reason none is reached: messaging 68 – the generated per-entity registrars, history-provider registrars and the component registrar, which are NATS subscription wiring and need a broker, so they belong to V04 rather than the unit suite. domain 42 – the generated *_table_io.cpp family, 26 of them, whose headers nothing includes; that is the B06 finding above, measured. repository 24 – the entity, mapper and repository triples for the eleven entities whose repository tests do not exist, against 16 test files for 27 repositories. presentation 19 – the generated history field mappers, exercised through the shell's history commands rather than in a unit. service 13 – the generated per-entity services for those same eleven entities. generators 8 – the synthetic generators with no case in generators_tests.cpp, which has 24 cases for 26 sources. app 2 and main.cpp 1 – the service entry point, which boots a process and a NATS connection and is what V03 and V04 exercise. The method is static reachability, not branch coverage, so a NATS-only path is invisible to it and "touched" does not mean exercised; this list is the coverage work item. |
projects/ores.codegen/scripts/survey_test_reachability.py --component dq --list |
5. Component Architecture Audit record
Run on 2026-09-27 against the tree at feature/dq-clean-batch-5, for H01.
The audit checks a component's shape, folders, naming, tests, documentation
and diagrams; the items that own the middle four are recorded above, and
this section is the record the item asks for.
5.1. Shape and folders
ores.dq is a composite of three parts, each with include/, src/ and
tests/, and each with its own component_overview.org and diagram. The
root carries the component model and the group index. No stray artefact was
found and no folder needed moving.
5.2. Naming, comments, documentation, tests
- Comments: H03, fixed. The component's one end-of-line comment and its seven narration comments were corrected, and a sweep of all six source languages finds no commented-out code.
- Documentation: H05, fixed.
socibecamesqlgen, and the root index is recorded as a group-level exemption. - Tests: V07 fixed and V08 surveyed. 0 of 551 cases are vacuous; 154 of 331 source files are reached by a test, and the untouched list is the coverage work item recorded on this task.
5.3. Diagrams
ores.dq.api, ores.dq.core and ores.dq.service each carry a .puml
with its rendered .png beside it. Both passes were run for the first time
in this pass:
- Automated.
build/scripts/generate_component_puml.pyrewrites everything above the manual sentinel from the component's headers. It had never been run against dq, so the committed diagrams were far behind: api 208 boxes to 641, core 88 to 140, service 4 to 7. It also stops the api diagram drawingsave_catalog_requestand the other names the protocol migration retired. - Manual. Every manual section was empty. Each now carries the
relationships the automated pass cannot see: 183 protocol-to-domain
associations in api, 56 service-to-repository associations in core, and 2
in service, using
*--for a data member and..>for a type that appears only in a signature, per the class-diagram conventions. Types from other components are left out. - Rendered and read. All three render clean with PlantUML and the images were read at full size, not merely regenerated.
Finding, recorded not fixed. Nothing runs the generator, so a diagram cannot stay current: running it across the tree changes 70 of 95 component diagrams, and every component that passed the clean standard did so with a stale or absent one. The gate is the fix, and it needs the tree regenerated first. Filed as a capture.
5.4. Complexity
lizard over projects/ores.dq/*/include and projects/ores.dq/*/src.
Generated sources are 702 of the component's 786 C++ files, so these are a
template signal rather than a component signal, as the audit says.
| Part | NLOC | Functions | Avg CCN |
|---|---|---|---|
| api | 8061 | 214 | 1.4 |
| core | 30394 | 1580 | 2.5 |
| service | 1016 | 37 | 1.2 |
| total | 39471 | 1831 | 2.4 |
ores.shell.dq, the shell adapter, is 14508 NLOC over 325 functions at an
average CCN of 6.7, which is the highest in the component's surface.
Top file by average CCN: core/src/service/publish_service.cpp, 7.4 average
over 9 functions and 2987 NLOC. Two functions cross lizard's warning
threshold and are recorded as findings rather than touched here:
publish_service::resolve_publication_order (CCN 18) and
report_definition_template_handler::list (CCN 17). Both are hand-written.
5.5. Triage
- Pass. Shape, folders, naming, comments, documentation, tests, diagrams.
- Trivial, fixed here. The stale diagrams and their empty manual sections,
the
socidependency line, the retired-topic constants. - Finding, record only. The two complexity hotspots above; the generator having no gate; the seven unmodelled artefact tables, which belong to pass 1.
6. Measurements
| Figure | Value | Command |
|---|---|---|
| Registered dq subjects | 210, plus 6 publish-from-dq triggers | grep -rhozE 'nats_subject[[:space:]]*=[[:space:]]*"dq\.v1\.[a-z0-9._-]+"' projects/ores.dq/api/include |
| Shell recipe inventory | 248 recipes in 181 categories | regenerate_shell_recipe_inventory.py --check |
| Shell root names | 214, 0 collisions | check_shell_menu_collisions.py |
| Handler permission codes | 226 across 7 components | check_handler_permissions.py |
| Protocol headers twinned | 77 across 10 components | check_protocol_twin_coverage.py |
| Models bound to a profile | 181 of 332, 14 profiles | check_model_drift.py --summary |
| Codegen suite | 924 passed | python -m pytest projects/ores.codegen/tests |
| dq test cases | 551, of which 0 vacuous (was 71) | projects/ores.codegen/scripts/survey_vacuous_tests.py --component dq |
| dq source files reached by a test | 154 of 331 | projects/ores.codegen/scripts/survey_test_reachability.py --component dq |
dq table_io headers with no consumer |
23 of 26 (tree-wide 183 of 221) | git ls-files 'projects/*/api/include/*/domain/*_table_io.hpp' |
| dq models with a dead presentation property | 21, 213 lines removed | regeneration byte-identical, so nothing read them |
| Models carrying them elsewhere | 75 in 7 components, 321 lines | grep -rn ':save_request_class:' projects/*/modeling/*.org |
| dq class-diagram boxes | api 208 committed, 641 generated | build/scripts/generate_component_puml.py --project ores.dq.api |
7. Open items
The following are recorded rather than closed, and each names where it goes next.
- H01 — the architecture audit record and the manual pass below each diagram's sentinel. Needs the rendered image read, so it wants a focused pass.
- V08's untouched list — the 177 source files no test reaches, per the survey above. The item asks for the list, and the list is recorded; closing it is the coverage work item and belongs in its own task.
- Pass 1's artefact tail — seven orphan artefact tables without a model or a recorded reason, and seven hand-written
*_artefact_create.sqlfiles that are neither generated nor deleted. Recorded on the pass 1 task, not here. - The H01 generator gate — nothing runs
build/scripts/generate_component_puml.py, so a component diagram is current only when someone remembers. Filed as a capture.
8. Notes
9. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
10. PRs
| PR | Title |
|---|---|
| #2316 | [agile] Close the dq clean-standard task |
| #2315 | [dq] Make the FSM state flags boolean |
| #2309 | [dq] Verify V03 and V04 against a live fleet |
| #2305 | [agile] Close the dq V08 item with a reason for every untouched file |
| #2304 | [dq] Pass H01: run both diagram passes and write the architecture audit |
| #2303 | [agile] Correct the dq H01 finding: the diagram generator exists |
| #2302 | [dq] Remove the model properties that name the retired message types |
| #2300 | [codegen] Add the vacuous-test and reachability survey tools |
| #2298 | [agile] Point the dq task at the branch that carried its last batch |
| #2296 | [dq] Assert what the dq tests actually observe |
| #2291 | [dq] Close the first clean-standard defect batch |
| #2286 | [dq] Admit ores.dq to the components under test |
11. Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|
12. Result
The task is complete in substance: every one of the 49 items passes, and the item table above records each one's result with the command or file that proves it. Five pull requests carried it — #2286, #2291, #2296, #2309 and this one.
Landed on feature/dq-m08-flags:
- M08 is fixed.
ores.dq.fsm_state'sis_initialandis_terminalwere integer columns with anin (0, 1)check while every consumer already read them as flags, so the storage side was the only thing that disagreed with the code; both are boolean, the two check constraints are gone, and the two hand-writtenores.reportingpredicates and five populate scripts are corrected. The item's own task also named the dq portfolio artefact'sis_virtual, integer against a boolean store with a conversion in the publish function; it is boolean too, the seeder writestrue=/=false, and four trade test scripts that inserted0into the refdata boolean column are corrected. Each change carries a one-shot migration inprojects/ores.sql/migration/, becausecompass db recreateonly helps a database built after the change. - M08 was verified against the running system, not only the build. The database recreated from the new schema, the fleet came up 25 of 25, and the two units that consume the flags —
ores.reporting.serviceandores.workflow.service— are among them. The first attempt at that failed: the services readis_initialwithstolland died onFailed to parse field 'is_initial', which was a stalelibores.dq.core.sothat predated the regenerated entity header, not a defect in the change. Rebuilding cleared it. - And the flags round-trip through the wire as booleans. Against the live fleet,
fsm_states add 11111111-1111-4111-8111-111111111111 m08-bool-check true false system.new_record bool-checkanswers{"...","is_initial":true,"is_terminal":false,...}, andfsm_states listanswers 28 rows, every one of them reporting the two fields astrueorfalserather than as a number. That exercises the whole path the change touched: the shell's bool token, the protocol struct, the dq service, the repository'ssqlgenread, the boolean column, and the JSON reply.
Landed on feature/dq-v03-fleet:
- V03 and V04 pass. The database recreates from scratch and all 25 units run, so the fleet was verified rather than excepted: the recipe replay answers 181 of 229 and records the other 48 as destructive. Two environment faults had to be cleared, and the replay exposed 24 orphan library scripts and a harness that needs the checkout .env exported. All three are recorded above.
Landed on feature/dq-clean-batch-5:
- H01 passes. Both diagram passes were run for the first time. The automated pass had never been run against dq, so the committed diagrams were far behind; the manual sections were empty and now carry 241 relationships the automated pass cannot see. All three diagrams render clean and were read at full size. The audit record, including the lizard complexity measure, is in its own section above.
Landed in the batch on feature/dq-clean-batch-4:
- M09 is fixed for dq. Eleven scalar properties in the entity models' presentation drawer name retired message types or headers nothing reads. Removing all 213 occurrences from 21 models left every generated file byte-identical, which is the proof they were dead. The same keys in 75 models across seven other components, and the skeleton template that would hand them to a new model, are filed as a capture.
Landed in #2296, on feature/dq-clean-batch-3:
- V07, the vacuous-test item, is fixed and proven. A census of all 551 dq test cases found 71 that could pass while their subject did nothing:
write_single_Xandwrite_multiple_Xasserted onlyCHECK_NOTHROW,read_latest_Xonly emptiness and a length,read_all_X_versionsonly a length,read_nonexistent_Xonlysize() =0=, and sixteen generator and parser cases only a count or a truthiness. All 71 now assert a value the test itself set. The census re-runs at 0. The proof the item asks for was run: withcatalog_repository::writemade a no-op on both overloads, all seven catalog repository cases fail, where before the fix the two write cases passed. - V08, the coverage survey, is recorded. Static include reachability over 331 source files, seeded from 79 test sources, puts 154 touched and 177 not. The untouched list is the coverage work item and is recorded by directory.
- B06 gained a measured finding. 23 of dq's 26
*_table_io.hppheaders are included by nothing but their own.cpp, so theoperator<<they define is unreachable and the generic JSON overload is what callers get. Tree-wide the figure is 183 of 221, so the fix is cross-cutting; it is filed as a capture and recorded here rather than changed in this batch.
Landed in #2291, on feature/dq-clean-batch-2, as seven commits:
485a1212f4gives the api module the namespace headercoreandservicealready had.5d001d773efixes two generator defects at source. The protocol-dependent facet gate asked whether the run renderedores.cpp.protocolrather than whether the model enables it, which emptied every doc-only run and meant no component had ever produced a shell recipe. And the create archetype's primary-key block dropped a declared default in silence. Each fix carries a regression test, and each test is mutation-proven.df60ea85fcrestores the LEI summary responder, derives the publish-from-dq function name from its subject, seeds the permission codes the dq handlers check, bindsbadge_severityandsubject_areatosimple-lookup, sweeps the comments, retires five change-reason constants that name a retired taxonomy, and reconciles the messaging reference with the registered surface.0f8a368785points the subject-area synthetic generator at a seeded data domain, a latent defect the new eventing test exposed.42755a9b4cgivespublicationthe shell unit its CRUD surface needs.3593ea312erecords this pass and retires the two captures it closed.b8692288adande4d5a7befdrefresh twoores.reportingrecipes and their scripts, which the fixed gate had never rendered.
What is open, recorded in the item table and the list above it:
- The H01 generator gate — nothing runs
build/scripts/generate_component_puml.py, so a component's diagram is current only when someone remembers to run it. Filed as a capture. - Pass 1's artefact tail — seven artefact tables with no model and, for four of them, no recorded reason. Recorded on the pass 1 task.
- V08's coverage work item — the 177 source files no test reaches, each with its reason recorded. Writing tests for them is its own task.
What these passes changed about the component: dq regenerates with no drift at every address and across the whole registry, the codegen suite is 924 tests, the fifteen checks on #2291 are green, and all five of its test suites pass. The shell surface grew from 211 to 214 root menus and dq's recipes from 12 to 25, the thirteen additions being ones the models already implied. Its test suite no longer has a single case that can pass while its subject does nothing, and both diagram passes have been run for all three parts.