Story: Codegen facet: generate the DQ publish-from-dq pattern
Table of Contents
This page documents a story in Sprint 30, carried unfinished from Sprint 28 at close. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
1. Goal
Generate the mechanical part of the DQ publish-from-dq pattern from the
entity's own codegen model, and stop hand-writing it. The pattern is the
SECURITY DEFINER ores_<component>_publish_<plural>_from_dq_fn that reads
an artefact table and writes the owning service's tables.
DQ artefacts states its contract;
DQ Bundle Publication Pipeline has
the end-to-end walkthrough and the codegen value assessment this story acts on.
The scope is narrower than this story first claimed. The measurements below are the reason, and two things follow from them:
- Generate the functions that are genuinely the same. About 29 of the 57 are
one plain per-row upsert into one target table, honouring
mode, matched on the entity's natural key. That family is a template. - Do not generate the handlers. The eight
publish_from_dq_handlerfiles are already one generic handler per service, not one per entity. Fifty-seven generated handlers would be fifty-seven copies of something that is not copied. They should be unified instead.
2. Status
| Field | Value |
|---|---|
| State | BACKLOG |
| Parent sprint | Sprint 30 |
| Now | Not yet started. Re-cut on 2026-10-15 against the tree; the first draft described a tree that no longer exists. |
| Waiting on | The declaration unit (see Decisions). It is the one choice that has to be made before the rest of the tasks can be cut. |
| Next | Decide the declaration unit. |
| Last touched | 2026-11-08 |
3. Acceptance
- The declaration unit is decided and recorded, and it expresses the real relation: 122 models opt into an artefact table and 57 functions publish them, so one function covers many artefact tables.
- Enabling the facet on an entity model generates its publish function for the plain one-table per-row upsert family, with no hand-written SQL beyond the declaration.
- Regenerating one existing hand-written instance produces a semantic zero-diff
against the hand-written version, which is then deleted. Use
asset_classesorcountries;report_definitionsis a helper-delegating set-based document and is the wrong first instance. - The functions that do not fit the family are named and recorded as deliberately hand-written, with the reason, so the facet's boundary is a statement rather than an omission.
- The eight
publish_from_dq_handlerfiles are unified, or the reason they cannot be is recorded. Generating 57 handlers is explicitly not the answer. dq_artefact_types_populate.sql,entity_table_writes_baseline.jsonand the pgTAP publish tests are updated for whatever moves.
4. Measurements
Taken on main at a0408fb02d.
| What | Value | Command | ||
|---|---|---|---|---|
| Publish functions | 57 across 7 components | =grep -rhoE "create or replace function ores_[a-z_]+_publish_[a-z0-9_]+_from_dq_fn" projects/ores.sql/create/ \ | sort -u \ | wc -l= |
| Their spread | refdata 47; analytics, iam, reporting, synthetic 2 each; assets, marketdata 1 each | the same command, per file | ||
| A second convention | 6 x ores_dq_<x>_publish_fn, so 63 across 8 services |
grep -rhoE "ores_dq_[a-z_]+_publish_fn" projects/ores.sql/create/dq/ |
||
| Handlers | 8 per-service .cpp, one of which (ores.dq) has no from_dq_fn |
grep -rl publish_from_dq_handler projects/ |
||
| Models that opt into an artefact table | 122 | domain_entity_artefact_create enablements |
||
| Plain per-row upsert family | 27, plus 2 with a p_params filter |
a structural classifier over the 57 bodies | ||
| Distinct structural shapes | 25 across the remainder | the same classifier | ||
Validate mode |
33 of 57; the other 24 ignore it or never validate it | the same classifier |
5. Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Decide the publish declaration unit | BACKLOG | A publish function does not correspond to one entity: 122 models opt into an artefact table and 57 functions publish them, and curve_configuration alone publishes 31 artefact tables. Decide what a single declaration is — one per publish function, or a document that groups the artefact tables a publication writes — and record it, because no other task in the story can be cut until it is settled. |
6. Decisions
- The declaration unit is not decided, and it is the first task. A publish
function does not correspond to one entity:
curve_configurationpublishes 31 artefact tables and analytics' today's market publishes 5, while 122 models opt into staging. The facet has to express "these artefact tables are published together by this function" before anything else can be cut. - The C++ half is unification, not generation.
subject_to_fnis already generic in every handler, and synthetic's validates against an allow-list. What differs is the subject spelling and the allow-list, not the handler. - This story no longer retires the
lookup_entityshadow models. The DQ artefact migration already did: no#+type: ores.codegen.lookup_entitymodel is left, and thesql_schema_artefact_createtemplate is deleted. - The
model_type ='schema'= enrichment bug is moot. It was raised as a prerequisite for rendering from alookup_entitymodel. That path is gone. - Subject constants are partly modelled already.
ores.codegen.operationrenders C++ protocol fromores.analytics.publish_operations.org,ores.iam.account_messages.organdores.reporting.operations.org; refdata, assets, dq and marketdata still hand-spell subjects inregistrar.cpp. A facet should use the operation models where they exist rather than add a fourth way to name a subject.
7. Out of scope
- The artefact table itself. That half is done (ores.sql.schema.domain_entity_artefact_create).
- The bundle and dataset registration rows, which are one-liners.
- The ~28 functions that are genuinely bespoke: helper-delegating documents,
multi-table publishes, and party-scoped publishes with their own
p_paramscontract.