Task: DQ-publish the calendar reference data

Table of Contents

This page documents a task in the Finish integrating calendar reference data: XML export and DQ publish story. It captures the goal, current status, acceptance, and any notes or results.

Goal

Wire the remaining calendar-adjacent reference-data tables — currency_country, currency_calendar, and currency_pair_convention_calendar — into the DQ bundle publication pipeline (see DQ Bundle Publication Pipeline), so this data reaches target services as dataset bundle publications, the same way currency_pairs is published today. calendar and calendar_type already have this support, delivered early as part of Model the calendar entity for codegen (refdata.v1.calendar-types.publish-from-dq, refdata.v1.calendars.publish-from-dq) — leaving both junction tables without a DQ publish path was flagged during the currency-calendar FK task's review as a gap that shouldn't be left unaddressed: every other reference-data table in this story is DQ-publishable, and the junctions inheriting "no DQ publish" from each other rather than from the fully-supported calendar pattern is not something to let stand. Since the publish_from_dq codegen facet described in the open capture has not been built yet, this follows the existing hand-written pattern: artefact table(s), publish-from-dq SQL function(s), and NATS handler wiring, per entity.

Status

Field Value
State DONE
Parent story Finish integrating calendar reference data: XML export and DQ publish
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-07-23

Acceptance

  • [X] Artefact tables (and matching drop scripts) exist for currency_country and currency_calendar, mirroring dq_calendars_artefact_tbl's shape.
  • [X] publish-from-dq SQL functions exist for both junctions in refdata_publish_from_dq_create.sql, supporting the same upsert/insert_only/replace_all mode contract as every other publish-from-dq function.
  • [X] Both are registered as NATS subjects (refdata.v1.currency-countries.publish-from-dq, refdata.v1.currency-calendars.publish-from-dq) in registrar.cpp.
  • [X] Both artefact types are registered in dq_artefact_types_populate.sql.
  • [X] currency_pair_convention_calendar is evaluated separately: it does not yet exist as a table (calendars are still attached to currency_pair_convention.advance_calendar as free text, per the pending advance_calendar migration task) — DQ publish for it is out of scope here and belongs with that migration once the column becomes a real relationship. Turned out moot: see Notes — it was already fully wired as part of task 1's delivery.
  • [X] Published bundles are visible and consumable via the Librarian UI. Verified via Verify currency_countries DQ publish: PASSED.

Plan

(Implementation strategy. Written when work starts; key decisions are distilled into the parent story's * Decisions at close, but the plan itself stays — it is the historical record of what we did.)

Checked what already existed before writing anything: currency_calendar and, surprisingly, currency_pair_convention_calendar both already had a full DQ-publish path (artefact table, drop script, publish-from-dq SQL function, NATS subject registration, artefact-type row) — delivered alongside the calendar=/=calendar_type pipeline in 0a167d548 [sql/dq] Add DQ publish pipeline for calendars and calendar types, before this task's acceptance criteria were written. Only currency_country had a real gap. Followed the existing hand-written pattern exactly (currency_calendar's artefact table/function as the template, since it is the closest shape: a simple two-column junction):

  1. dq_currency_countries_artefact_create.sql / dq_currency_countries_artefact_drop.sql, wired into dq_create.sql=/=dq_drop.sql.
  2. ores_refdata_publish_currency_countries_from_dq_fn in refdata_publish_from_dq_create.sql.
  3. refdata.v1.currency-countries.publish-from-dq added to the subject list in registrar.cpp (generic publish_from_dq_handler derives the SQL function name from the subject, so no C++ dispatch logic needed).
  4. currency_countries artefact-type row in dq_artefact_types_populate.sql (display_order 62).

Registering the artefact type alone isn't enough for the Librarian to actually offer it — a bundle only publishes datasets it's a member of, and a dataset needs its own ores_dq_datasets_tbl row plus real rows in the shadow artefact table. So, mirroring refdata_currency_calendars_seed_populate.sql:

  1. refdata_currency_countries_seed_populate.sql: registers the refdata.currency_countries dataset and copies ores_refdata_currency_countries_tbl (already seeded by the foundation layer) into ores_dq_currency_countries_artefact_tbl verbatim — no derivation needed here, unlike currency_calendars, since currency_countries is the source of truth, not something derived from it.
  2. refdata.currency_countries added to the base bundle's members in dq_dataset_bundle_member_populate.sql at display_order 18 (after currency_pair_convention_calendars at 17, to avoid renumbering an already-shipped entry).

compass db recreate confirmed the new SQL applies cleanly end to end: 39 currency-country rows land in the DQ artefact table, and calling ores_refdata_publish_currency_countries_from_dq_fn directly against the real seeded dataset successfully republishes all 39 rows (same live-table upsert path the Librarian's workflow step would invoke). Calling it with a bogus dataset id raises the same "Dataset not found" error as every sibling function. Full rat (build + test) passed clean on both the pre- and post-seed-script db recreate.

Notes

currency_calendar's and currency_pair_convention_calendar's DQ publish infrastructure was already complete when this task started — this task's own acceptance criteria (written 2026-07-13/21) were already stale by the time work began. Left a note here rather than silently rewriting history, since the "not yet a table" claim about currency_pair_convention_calendar was true when written but the table, and its DQ publish path, landed in the interim (task 1's early delivery + this story's junction-persistence follow-up, PR #1679).

Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario, run through the QA Validation Runner panel) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
Verify currency_countries DQ publish PASSED Librarian bundle publish + Publication Audit dialog; cross-checked against the database (39 rows, no dupes, EUR maps to all 6 Eurozone countries).

PRs

PR Title
#1681 [refdata,sql] DQ-publish currency_country into the base bundle

Review

# Comment summary File Decision Notes
1 display_order (62) placed before currency_pair_convention_calendars (61) in file order dq_artefact_types_populate.sql Accepted Reordered the INSERT rows to match numeric display_order
2 Missing blank line before "* Out of scope" heading story.org Accepted Added blank line for consistency with other heading transitions

Result

currency_country now has a full DQ-publish path: artefact table (ores_dq_currency_countries_artefact_tbl), drop script, ores_refdata_publish_currency_countries_from_dq_fn, a refdata.v1.currency-countries.publish-from-dq NATS subject, a currency_countries artefact-type row, a dataset-registration + seed-populate script mirroring the live table, and base bundle membership at display_order 18. currency_calendar and currency_pair_convention_calendar turned out to already be fully wired (see Notes) — only currency_country had a real gap.

Verified end to end: full rat build+test clean; a real barclays_system_provision.ores run publishing the base bundle against the Barclays Plc tenant completed refdata.currency_countries without error; the Publication Audit dialog and a direct database check both confirm 39 currency-country rows landed with zero duplicates and the correct EUR/Eurozone mapping. Manual QA scenario Verify currency_countries DQ publish: PASSED.

Filed a follow-up capture, Add a currency-country assignment widget to CurrencyDetailDialog, for the UI gap discovered while writing the scenario: currency_calendar has a CalendarAssignmentWidget in the currency detail dialog; currency_country has no UI anywhere. Out of scope for this task.

Emacs 29.3 (Org mode 9.6.15)