DQ artefacts

The artefact belongs to the model of the data it stages. A book's artefact table comes from ores.refdata.book, a currency's from ores.refdata.currency, an image's from ores.assets.image: one model states the entity and its staging shape, and the two cannot drift.

1. Opt in on the owning model. Add :ores.sql.schema.domain_entity_artefact_create.enabled: true to the model's properties. The archetype is ores.sql.schema.domain_entity_artefact_create; it is off by default because most entities have no import.
2. Declare the staging shape when it differs. By default the artefact table projects the entity's columns. Add an * Artefact columns section, one ** <column> per staging column in order, when the import needs something the store does not: a natural key where the store holds a uuid, a field the publisher supplies left out, or a denormalised row that publishes into several tables. The first column is the staging key unless :key: names another. An expression table in the section's own body, above the first column, states the staging table's check constraints, as the ** Checks table of the * SQL section does for the entity's.
3. Regenerate the SQL schema for the model: ./compass.sh codegen regenerate --component <component> --address ores.sql.schema --entity <entity>. The table lands in projects/ores.sql/create/dq/dq_<plural>_artefact_create.sql.
4. Include it by hand. Add the \ir line to dq_create.sql, and write the drop script projects/ores.sql/drop/dq/dq_<plural>_artefact_drop.sql with its line in dq_drop.sql, as every artefact does today.
5. Seed the rows. A populate script in the owning component's populate folder registers the dataset with ores_dq_datasets_upsert_fn, naming its catalogue and its artefact type, and inserts the artefact rows for that dataset id. It is idempotent: it leaves a dataset that already has rows, or replaces them.
6. Name the artefact type. Add a row to projects/ores.sql/populate/dq/dq_artefact_types_populate.sql: the code, the artefact table, the target table and the publish subject, <component>.v1.ops.publish_<plural>_from_dq. A new kind of data gets a new artefact type; data of an existing kind reuses its type.
7. Put the dataset in a bundle with ores_dq_dataset_bundle_members_upsert_fn. system_core holds the installation's own data and is published into the system tenant once; every other bundle is published into each tenant a profile names.
8. Declare what it depends on with ores_dq_dataset_dependencies_upsert_fn. The publication orders datasets by their dependencies and nothing else: the display order of a bundle member does not decide it. A dataset whose rows validate against another dataset's rows, such as a feed naming its provider, declares that dataset.
9. Publish in the owning service. The service subscribes to the subject in its registrar and lets its publish_from_dq_handler call ores_<component>_publish_<plural>_from_dq_fn. Follow the component's existing handler: synthetic checks each function against an allow-list, and refdata lists its subjects in the registrar.
10. Write the publish function in projects/ores.sql/create/<component>/<component>_publish_from_dq_create.sql.
11. Test the publish in pgTAP.

Every publish function has the same signature and contract:

create or replace function ores_<component>_publish_<plural>_from_dq_fn(
    p_dataset_id uuid,
    p_target_tenant_id uuid,
    p_mode text default 'upsert',
    p_params jsonb default '{}'::jsonb
)
returns table (action text, record_count bigint)

• It is security definer, with set search_path = public, pg_temp, because it reads the DQ artefact table and writes the owning service's table.
• It accepts the modes upsert, insert_only and replace_all, and refuses any other. replace_all closes current rows before it writes the dataset's rows, and the function states the scope it closes: the rows this dataset wrote, or every row of the table in the target tenant or party. The two differ when several datasets write one table, so a function that closes the wider scope says so in its comment. It returns one row per action it took: inserted, updated, skipped or deleted, each with its count.
• It is idempotent. A repeat publish of the same dataset changes nothing it already wrote. A function that upserts compares a row it holds with the staged row and skips the row when the values match, so a repeat publish adds no version. The older functions rewrite every row; do not copy that.
• It closes a row with clock_timestamp(), as the tables' triggers and delete rules do, never with current_timestamp: a row written earlier in the same transaction would otherwise get a valid_to before its valid_from.
• Installation data refuses any target but the system tenant. Reference data that every tenant reads, such as the ORE parameters, is a member of both system_core and base, so its function writes to the system tenant and to each tenant. That doubles the data on purpose until readers use one source; see One source for reference data that the system and each tenant read.
• It reads p_params only for what the bundle publication passes, such as the party a party-scoped publish writes for.
• It writes the owning tables directly, so it is listed in the entity-writes baseline, projects/ores.codegen/scripts/entity_table_writes_baseline.json, beside the other publish functions.

• [ ] The owning model carries the artefact flag, and an * Artefact columns section only when the staging shape differs.
• [ ] No ores.dq model, DQ C++ code, shell command or TypeScript was added for the artefact.
• [ ] The generated create script is included in dq_create.sql, with its drop script in dq_drop.sql.
• [ ] The dataset is registered with its catalogue and artefact type, and its rows are seeded idempotently.
• [ ] The artefact type names the artefact table, the target table and the subject.
• [ ] The dataset is in the right bundle: system_core for installation data, a tenant bundle otherwise.
• [ ] Every dataset it validates against is declared as a dependency.
• [ ] The owning service subscribes to the subject, and its handler reaches the function.
• [ ] The publish function follows the contract above and is in the entity-writes baseline.
• [ ] A pgTAP test publishes the dataset, publishes it again in upsert, insert_only and replace_all in the same transaction, and checks the rows.

The hand-written pattern still exists in the tree and is not to be copied. The last group is undecided rather than legacy: where DQ owns the data, as with the badges and coding schemes, the ores.dq model is the owning model and follows the one way. Each group has a task in Bring every DQ artefact table onto the canonical model.

Two patterns are retired. A separate ores.dq model that duplicated a domain model of another component, with a DQ C++, shell and TypeScript stack for the staging table, went first: the CSAs, netting agreements, netting sets, report definitions and market data providers now come from their owning models, by Move the duplicated DQ models onto their owners. Five hand-written tables also moved onto their owning models: the CRM topology, the market data observations and the three synthetic feed tables, by Model the hand-written artefact tables.

The older lookup template, sql_schema_artefact_create, is gone. Its fifteen tables – trading_account_types, asset_classes, asset_measures, benchmark_rates, business_centres, business_processes, cashflow_types, entity_classifications, local_jurisdictions, party_relationships, party_roles, person_roles, regulatory_corporate_sectors, reporting_regimes and supervisory_bodies – are generated from their own models' artefact hook, and the template and its literate source are deleted, by Move the lookup artefact tables onto domain models.

An ores.dq model whose owner is unsettled is settled by one rule: the owning model is the model of the entity the artefact stages, in the component that owns the published rows. Where the artefact's own type row is terminal – its target_table and target_subject both null – the artefact table is the entity's only store, and DQ owns the model. The four DQ registries (badge_definition, badge_severity, code_domain, coding_scheme) and the three GLEIF staging models (lei_entity, lei_relationship, lei_bic) are terminal and stay in ores.dq. Where the type row publishes onward, the rows are the publishing component's data in transit, and the staging shape moves onto that component's model as an * Artefact columns section: counterparty_alias onto ores.refdata.counterparty_identifier, netting_set_alias onto ores.refdata.netting_set_identifier, and synthetic_fx_spot_config onto ores.synthetic.fx_spot_generation_config. The report configuration binding staged in dq_report_configurations_artefact_tbl moved the same way, onto ores.reporting.report_configuration, by Settle the owner of the DQ-only models.

Junction artefact tables come from their own template, sql_schema_junction_artefact_create, and are out of scope of this page.

The eight publish_from_dq_handler files, one per service, repeat one pattern by hand and validate the subject in different ways. A new artefact follows its component's handler; making them one is not yet a task.

The artefact table lives in the dq schema, but it is the owning component's data in transit. That is why the owning model generates it, why the owning service publishes it, and why DQ holds no code for it: DQ owns the datasets, bundles, artefact types and publications, not the rows they carry.