DQ Bundle Publication Pipeline

┌─────────────────────────────────────────────────────────────────────┐
│ Librarian UI (went with the removed desktop client)                  │
│  user picks a bundle, fills per-artefact-type params, clicks Apply  │
│  → dq.v1.bundles.publish  {bundle_code, target_tenant_id, params}   │
└───────────────────────────┬─────────────────────────────────────────┘
                            │  NATS request
┌───────────────────────────▼─────────────────────────────────────────┐
│ ores.dq.core (publication_service / publication_handler)            │
│  loads bundle members in display_order from                        │
│  ores_dq_dataset_bundle_members_tbl + ores_dq_artefact_types_tbl    │
│  → builds bundle_publish_workflow_request                          │
│  → starts "bundle_publish_workflow" via the workflow engine         │
└───────────────────────────┬─────────────────────────────────────────┘
                            │  one workflow step per dataset
┌───────────────────────────▼─────────────────────────────────────────┐
│ ores.workflow engine                                                 │
│  dispatches each step's command_subject in display_order,          │
│  halts the saga on the first step failure (no compensation)        │
└───────────────────────────┬─────────────────────────────────────────┘
                            │  <service>.v1.<entity>.publish-from-dq
┌───────────────────────────▼─────────────────────────────────────────┐
│ Target service (ores.refdata.core / ores.assets.core / ...)         │
│  publish_from_dq_handler → ores_<service>_publish_<entity>_from_dq_fn│
│  (SECURITY DEFINER: reads ores_dq_<entity>_artefact_tbl,            │
│   writes only ores_<service>_* tables)                              │
└─────────────────────────────────────────────────────────────────────┘

• ores_dq_dataset_bundles_tbl — one row per bundle (code, name, description). Codegen entity model: projects/ores.dq/modeling/ores.dq.dataset_bundle.org.
• ores_dq_dataset_bundle_members_tbl — junction: which datasets belong to a bundle, and in what display_order. Codegen junction model: projects/ores.dq/modeling/ores.dq.dataset_bundle_member_junction.org.
• ores_dq_artefact_types_tbl — lookup table mapping an artefact-type code (e.g. gleif.lei_counterparties.small) to its artefact_table (source DQ table), target_table, and target_subject (the NATS subject the workflow step will call). Codegen lookup-entity model: projects/ores.dq/modeling/ores.dq.artefact_type_lookup_entity.org.
• ores_dq_<entity>_artefact_tbl — the shadow staging table for one artefact type: same columns as the target entity, plus dataset_id and tenant_id bookkeeping, but no PK/FK constraints (a row can be incomplete or reference an entity that doesn't exist yet in the target service). This is also codegen-generated, from its own ores.codegen.lookup_entity model living alongside the entity type in projects/ores.dq/modeling/ — e.g. ores.dq.report_definition_lookup_entity.org drives sql_schema_artefact_create.mustache to produce ores_dq_report_definitions_artefact_tbl (projects/ores.sql/create/dq/dq_report_definitions_artefact_create.sql, header confirms AUTO-GENERATED FILE). Flags on the lookup-entity model (has_coding_scheme, has_image_id, has_artefact_insert_fn) control optional bookkeeping columns/functions.

So four of the five metadata/schema artefacts are codegen-generated today — bundle, junction, artefact-type lookup, and the per-entity shadow artefact table. What is not generated is the data that fills the shadow table (next) and the publish function that drains it (Stage 4).

Once the shadow table exists, it still needs rows. This splits into two genuinely different cases, not one:

publication_service::publish_bundle (or the equivalent bundle/dataset handler path, see ores.dq.core/src/messaging/publication_handler.hpp and dataset_handler.hpp) loads the bundle's members in display_order, builds a bundle_publish_workflow_request (one bundle_publish_workflow_dataset per member, carrying dataset_id, target_subject, mode, params_json), and starts a bundle_publish_workflow instance.

The workflow definition itself (projects/ores.dq.api/include/ores.dq.api/workflow/bundle_publish_workflow.hpp, register_bundle_publish_workflow) builds one workflow_step_def per dataset: command_subject is the dataset's target_subject, the command payload is a publish_from_dq_command (dataset_id, tenant_id, mode, params_json), and there is no compensation step (publish functions are idempotent upserts, so a failed saga simply halts and leaves already-published datasets in place). Progress is tracked via the standard workflow step-completed events; a final audit row is written to ores_dq_bundle_publications_tbl (projects/ores.sql/create/dq/dq_bundle_publication_create.sql).

Each target service exposes a NATS subject <service>.v1.<entity>.publish-from-dq, handled by a publish_from_dq_handler that calls a SECURITY DEFINER SQL function ores_<service>_publish_<entity>_from_dq_fn defined in that service's own SQL tree, e.g. projects/ores.sql/create/refdata/refdata_publish_from_dq_create.sql (22 functions), assets_publish_from_dq_create.sql (1), reporting_publish_from_dq_create.sql (1). This is the one sanctioned exception to strict service-table isolation: the function reads ores_dq_<entity>_artefact_tbl (cross-service read, DDL-owner privilege) and writes only to tables owned by its own service — DQ itself never gets DML on another service's tables. See doc/plans/2026-05-14-dq-publish-pattern.org for the full rationale and migration history behind this rule.

Nothing about this stage is codegen-generated — each publish_..._from_dq_fn is hand-written SQL, and each publish_from_dq_handler registration is hand-written C++, repeated per entity.

DataLibrarianWindow, in the desktop client since removed, listed bundles ("The Stacks") and their member datasets ("The Registry"). Selecting Publish opened PublishBundleWizard, which walked the user through per-artefact-type parameter pages before sending dq.v1.bundles.publish. Parameter pages are hand-coded per artefact type today (e.g. LeiPartyConfigPage for LEI's root_lei etc.) — the originally-intended generic publication_params_schema (a jsonb column on ores_dq_artefact_types_tbl that would let the wizard render config UI from a schema instead of bespoke pages) is not yet implemented; see doc/agile/product_backlog/next/add_publication_params_schema_to_artefact_types.org.

Adding a brand-new bundle/artefact type currently means hand-writing, per artefact type:

1. The populate/ETL script (stage 2) — unless the source is FPML coding-scheme data, in which case fpml_parser.py already handles it generically.
2. The target service's publish_..._from_dq_fn SQL function (stage 4).
3. The publish_from_dq_handler registration in the target service's registrar (stage 4).
4. A wizard config page in PublishBundleWizard if the artefact type needs user-supplied parameters (stage 5).

The bundle row, junction row, artefact-type lookup row, and the shadow artefact table itself (stage 1) are all already backed by codegen — and so is the populate step for FPML-shaped sources. The publish function is the one gap with no codegen path today, for any source: any effort to add a new bundle type at scale (e.g. a synthetic-market-data bundle) needs either to accept the hand-written cost of one more publish_..._from_dq_fn=/handler pair, or to extend =ores.codegen with a reusable "publish-from-dq" profile before proceeding — the 25 existing functions are structurally identical enough (read artefact table, honor mode, upsert into one target table) to make a generic profile plausible.

Every file the pipeline touches for one concrete artefact type, in pipeline order:

1. Bundle row — projects/ores.sql/populate/dq/dq_dataset_bundle_populate.sql registers the risk_management bundle itself (ores_dq_dataset_bundles_upsert_fn('risk_management', 'Risk Management', ...)) — the thing a user picks in the Librarian.
2. Bundle membership — projects/ores.sql/populate/dq/dq_dataset_bundle_member_populate.sql maps the ore.report_definitions dataset into the risk_management bundle at display_order 40 (after business_units/portfolios/books at 10/20/30) — this row is what stage 3's workflow builder iterates to build one step.
3. Artefact-type registration — projects/ores.sql/populate/dq/dq_artefact_types_populate.sql registers the report_definitions artefact type, pointing artefact_table → dq_report_definitions_artefact_tbl, target_table → reporting_report_definitions_tbl, and target_subject → reporting.v1.report-definitions.publish-from-dq — this row is the glue the workflow step uses to know which NATS subject to call (stage 1).
4. Shadow-table codegen model — projects/ores.dq/modeling/ores.dq.report_definition_lookup_entity.org (ores.codegen.lookup_entity) declares the staging-table shape (name, report_type, schedule_expression, …, display_order, artefact indexes) that generates the next file.
5. Generated shadow table — projects/ores.sql/create/dq/dq_report_definitions_artefact_create.sql (auto-generated from #4) creates ores_dq_report_definitions_artefact_tbl — no PK/FK constraints, just dataset_id=/=tenant_id bookkeeping plus the entity's columns (stage 1).
6. Dataset registration + seed data — projects/ores.sql/populate/reporting/reporting_report_definitions_populate.sql registers the ore.report_definitions dataset (ores_dq_datasets_upsert_fn) and inserts the 28 standard ORE analytics report rows straight into ores_dq_report_definitions_artefact_tbl (stage 2). This is a "seed-time inversion" per DQ Publish Pattern: reporting's own populate script writes into a DQ-owned table, but only at DDL-owner setup time, not at runtime — distinct from the publish-time cross-service write in #9 below.
7. Target entity codegen model — projects/ores.reporting/modeling/ores.reporting.report_definition.org (ores.codegen.entity) is the "real" report_definition domain type — unrelated to publishing, but it's what #8's target table is generated from, and what #9 ultimately writes rows shaped like.
8. Target table — projects/ores.sql/create/reporting/reporting_report_definitions_create.sql creates ores_reporting_report_definitions_tbl, the real, constrained, party-scoped table that end users query.
9. Publish function — projects/ores.sql/create/reporting/reporting_publish_from_dq_create.sql defines ores_reporting_publish_report_definitions_from_dq_fn (SECURITY DEFINER): reads ores_dq_report_definitions_artefact_tbl, honors mode (upsert/insert_only/replace_all), writes only into ores_reporting_report_definitions_tbl (stage 4). Hand-written — the gap identified above.
10. NATS handler — projects/ores.reporting/core/src/messaging/publish_from_dq_handler.cpp (declared in the matching .hpp) receives a publish_from_dq_command and calls #9's SQL function (stage 4). Hand-written per service, not per entity, but still not generated.
11. Registrar wiring — projects/ores.reporting/core/src/messaging/registrar.cpp subscribes #10 on the subject reporting.v1.report-definitions.publish-from-dq named in #3 (stage 4).
12. Bundle-publish orchestration (generic, not report_definitions-specific) — projects/ores.dq/core/include/ores.dq.core/messaging/publication_handler.hpp turns the bundle's member list into a bundle_publish_workflow_request and starts the workflow (stage 3); projects/ores.dq/api/include/ores.dq.api/workflow/bundle_publish_workflow.hpp is the workflow definition that turns that request into one step per dataset, dispatching to #3's target_subject (i.e. #11's subscription).
13. Publish entry point (generic — no bespoke wizard page needed here, unlike LEI's root_lei parameters). The UI lists the risk_management bundle for the user to select, then sends dq.v1.bundles.publish with bundle_code = "risk_management" on Apply. No per-entity code is needed, since report_definitions needs no publish-time parameters (stage 5). The screen that did this, DataLibrarianWindow, went with the desktop client, so nothing sends the request today.

Rating each of the 13 worked-example files/artefacts by where codegen already pays off, where it plausibly could, and where it structurally can't — this is the reference to consult before adding a new type to the bundle system.

Recommendation: build a single new ores.codegen facet — provisionally publish_from_dq — that hangs off an entity's existing codegen model and, from one declaration, generates items #3, #4, #9, #10, #11. Once that facet exists, adding a new type to the bundle system reduces to: model the entity normally (already required for any entity) → enable the publish_from_dq facet → hand-write only the bundle/membership rows (one-liners) and the actual seed data.