Task: Move the duplicated DQ models onto their owners
Table of Contents
This page documents a task in the Bring every DQ artefact table onto the canonical model story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Each of the six artefact tables is generated from its owning model, and no duplicate ores.dq model or DQ stack remains.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Bring every DQ artefact table onto the canonical model |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-13 |
3. Acceptance
- The six ores.dq models and their generated stacks are deleted, after checking nothing still consumes them.
- Each owning model carries the flag; the populates, publish functions and artefact types use the regenerated tables.
- The datasets publish as before: pgTAP and an Acme provision show the same rows.
4. Plan
- Confirm the inventory. Five of the six tables come from a duplicate
ores.dqmodel.risk_report_configsdoes not: it already comes fromores.reporting.risk_report_config, which carries the flag. The guide's table was wrong about it. - For each of the five, set the flag on the owning model. Where the staging
shape differs from the entity, write an
* Artefact columnssection, and carry the DQ model's* Artefact indexesover. A script reads the current artefact table, so the section states the same columns, types, nullability and defaults. - Regenerate the SQL schema for each owning model and diff the artefact table against the old one. The table must not change, other than its comment header.
- Delete each
ores.dqmodel and its generated stack: the DQ API domain, JSON and table I/O, generator and protocol; the DQ core handler, registrar, entity, mapper, repository and service; and the generated TypeScript. Remove the registrar wiring, the shell command, the recipes, the protocol rows and the read permissions only these handlers checked. - Follow the provider table rename through the hand-written SQL.
- Run the drift checks, build, ctest and pgTAP. Recreate the database and provision Acme.
5. Measurements
Every number this task claims about a tree, with the command that produced it
and the commit it was measured at. A number without both is not a measurement,
because a clean-up record describes a tree and trunk moves under a branch: a
consumer count, a gate's component count and a suite's size all change with no
code changing, and the reader of a bare number cannot tell which tree it came
from. Where the raw output matters, put it in an evidence/ file beside the
task and name that file in the Command column.
| What | Value | Command | Commit |
|---|---|---|---|
| Branch diff | 161 files, 350 insertions, 13,409 deletions; 128 files deleted | git diff --shortstat origin/main...HEAD |
667112cd48 |
| Artefact table change | none, other than the folded tier columns and the provider table name | git diff projects/ores.sql/create/dq/ after regeneration |
f6677f1341 |
| ctest | 93 of 93 passed | compass check --only ctest |
667112cd48 |
| pgTAP | 45 files, 433 tests, PASS | compass check --only pgtap |
667112cd48 |
| pgTAP after rebasing on main | 46 files, 449 tests, PASS; build clean | compass db recreate, compass check --only pgtap, compass build |
823d6e8a44 |
| Codegen suite | 1,397 passed | pytest projects/ores.codegen/tests |
this commit |
| Acme published rows | csas 35, netting_agreements 45, netting_sets 80, report_definitions 78 in the Acme tenant; 8 providers in the system tenant | compass db sql on the current-row counts |
667112cd48 |
| Acme feeds | 55 bindings for each of the 5 parties; ticks arriving | compass db sql on feed bindings and observations |
667112cd48 |
6. Notes
- The provider table is renamed. The owning model's plural is
market_data_providers, so the staging table is nowores_dq_market_data_providers_artefact_tbl. The populate, the publish function, the artefact type row, the drop script and the pgTAP test follow the rename. The dataset code and the artefact type code do not change. - The report definition tier columns move into the model. The tier and the
applicable jurisdiction were added by a hand-written alter script,
dq_report_definitions_artefact_tier_alter.sql. They are now the last two artefact columns ofores.reporting.report_definition, and the alter script is deleted. - The report definition template handler stays.
dq.v1.report_definition_templates.listreads the artefact table in SQL, not through the deleted repository, and the reporting service calls it. Its permission,dq::report_definitions:read, stays with it. - The DQ report definition capture is resolved. The capture BAFCC111 said the DQ report definition entity names a table that no script creates. The entity is gone, so the capture is filed with a resolution.
- The CSA, netting agreement and netting set tables keep their natural-key shape: a netting set code, an agreement number and a counterparty LEI where the store holds uuids. Publishing resolves them in the target tenant.
7. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
8. PRs
| PR | Title |
|---|---|
| #3219 | [dq] Move the duplicated DQ models onto their owners |
9. Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | The staging shim gate test named the deleted dq report definition model | test_staging_only_facets_gate.py |
Accept | The test builds a fixture shim with a rival plural; the gate stays. |
| 2 | Each artefact table comment now comes from the owning entity's description, not a staging description | generated dq_*_artefact_create.sql |
Decline | This is how the canonical template works for every table. The staging difference is stated in the model's Artefact columns prose. |
| 3 | dq::report_definitions:read names a resource that has no DQ entity now |
iam_permissions_populate.sql |
Decline | The template handler still checks it and ReportingService holds it. A rename changes roles for no gain in this task. |
| 4 | The drift checks also report drift from main: shell scripts, history registrars, ER diagrams, a new entity writer | various | Decline | Not this branch. Left out of the commits. |
10. Result
The five duplicated DQ models are gone. csas, netting_agreements and
netting_sets come from their refdata models, report_definitions from
ores.reporting.report_definition, and the provider staging table from
ores.synthetic.market_data_provider, now named
ores_dq_market_data_providers_artefact_tbl. Each owning model carries the
artefact flag and an * Artefact columns section where the staging shape
differs, and the regenerated tables keep their columns. The report definition
tier columns are in the model, and the alter script is deleted. The DQ API,
core, shell and TypeScript stacks of the five models are deleted, with their
registrar wiring, shell command, recipes, protocol rows and permissions.
risk_report_configs needed no change: it already came from its reporting
model, and the DQ guide now says so.
Acceptance: the models and stacks are deleted, and nothing consumed them other
than the wiring removed here. The owning models carry the flag. pgTAP, ctest and
an Acme provision pass, and the datasets publish into the Acme tenant. The
provision recorded one failed step, start_market_feeds, which timed out while
the feeds started. This branch does not touch that path; the capture
start_market_feeds_step_times_out_while_the_feeds_start records it.