Task: Bind ores.refdata entities to profiles; verify zero-diff regen

Table of Contents

This page documents a task in the Entity classification and drift baseline across all components story. It captures the goal, current status, acceptance, and any notes or results.

Goal

Refined from a read-only classification scanner into a concrete, falsifiable proof: implement real :profile: binding in the code generator, rebind every ores.refdata entity that matches one of the 5 named profiles (Variability Profiles) to that profile — removing its now-redundant, individually-authored feature properties — regenerate, and confirm byte-identical output against what was checked in before the rebind. A zero-diff regen is the strongest possible confirmation that the profile catalogue actually captures what it claims to: if binding to a profile changes what gets generated for a single entity, the profile (or the classification of that entity as matching it) is wrong, and that must be fixed before this pilot is trusted, let alone rolled out to the other ~19 components.

This is the "Profile match" classification dimension from the story's settled list, but validated far more strongly than a read-only report could: not "the feature values look equal on paper" but "codegen produces the same files whether the entity's variability is spelled out explicitly or expressed as one profile name."

Entities matching no profile are not a failure of this task — they are real data, either genuinely novel configurations (candidates for a 6th profile, once a second entity repeats the same combination) or entities whose per-entity features are correctly bespoke. Record them, don't force a match.

Status

Field Value
State DONE
Parent story Entity classification and drift baseline across all components
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-07-31

Acceptance

  • org_loader.py resolves a :profile: <name> property on an entity model by reading that profile's own variability_<slug>.org Assignments table as the source of truth (no hardcoded second copy of the profile catalogue) and merging its feature values in before the rest of loading proceeds; an explicit per-entity property still overrides its profile's value where both are present.
  • Every ores.refdata entity's current feature values are computed and compared against the 5 profiles; each is recorded as a match (which profile) or no-match.
  • Every matching entity is rebound: its explicit per-entity feature properties replaced with :profile: <name>, keeping only what the profile does not cover (structural fields, per-entity naming).
  • Every rebound entity is regenerated and diffed against its pre-rebind checked-in output. Zero diff for all of them is the pass condition; any non-zero diff is a bug in either the profile catalogue or the matching logic and must be fixed, not worked around, before this task is done.
  • currency (the fully-featured-lookup reference entity) is among the rebound and zero-diff-verified set.
  • No-match entities are listed with their actual feature values, not silently dropped.

Plan

(Implementation strategy. Written when work starts; key decisions are distilled into the parent story's * Decisions at close, but the plan itself stays — it is the historical record of what we did.)

Metatypes covered

ores.refdata/modeling/*.org (69 files) spans four metatypes; only two carry any variability :profile: binding can express:

Metatype Count :profile: binding
ores.codegen.entity 58 Yes — 6 profiles, 100% match, zero new work
ores.codegen.junction 9 Yes — added this task (7th profile)
ores.codegen.module 1 N/A — a catalogue index, not an entity
ores.codegen.component 1 N/A — component overview, not an entity

Worklist — process one entity at a time: remove its explicit

feature properties the target profile covers, add :profile:, regenerate at --address ores, diff against the pre-rebind checked-in output, revert and flag on any non-zero diff.

Non-codegened (2) — no variability applies, tracked not skipped

File Justification
module.org ores.codegen.module: the component's entity catalogue index, not itself a generated artefact.
component_overview.org ores.codegen.component: component-level metadata doc, not a generated artefact.

ores.codegen.junction (9) → tenant-scoped-junction

Entity Status
currency_calendar (reference entity) DONE
calendar_date DONE
currency_country DONE
currency_currency_group_junction DONE
currency_pair_convention_calendar DONE
party_counterparty_junction DONE
party_country_junction DONE
party_currency_junction DONE
tenor_convention_resolution DONE

ores.codegen.entity (58), grouped by matched profile

  • simple-lookup (34)
    Entity Status
    asset_class_code DONE
    book_purpose_type DONE
    book_status DONE
    business_centre DONE
    business_day_convention_type DONE
    calendar DONE
    calendar_type DONE
    contact_type DONE
    country DONE
    currency_group DONE
    currency_market_tier DONE
    currency_pair DONE
    currency_pair_classification DONE
    currency_pair_convention DONE
    curve_role DONE
    day_count_fraction_type DONE
    floating_index_type DONE
    instrument_code DONE
    ledger_feed_type DONE
    leg_type DONE (bound; missing has_change_reason_cache was a genuine gap, fixed — see Notes)
    monetary_nature DONE
    party_id_scheme DONE
    party_status DONE
    party_type DONE
    payment_frequency DONE
    purpose_type DONE
    regulatory_book_type DONE
    rounding_type DONE
    tenor DONE
    tenor_anchor DONE
    tenor_convention DONE
    tenor_kind DONE
    tenor_resolution_algorithm DONE
    tenor_unit DONE
  • fk-scoped-child (3)
    Entity Status
    book DONE
    business_unit NO MATCH (correctly bespoke — genuinely tenant-only, not workspace-scoped; fk-scoped-child would wrongly add a workspace_id column — see Notes)
    portfolio DONE
  • self-referencing-hierarchy (12)
    Entity Status
    business_unit_type DONE
    calendar_exception DONE
    calendar_rule DONE
    counterparty DONE
    counterparty_contact_information DONE
    counterparty_identifier DONE
    crm_driver_pair DONE
    crm_enabled_derived_pair DONE
    crm_topology_config DONE
    party DONE
    party_contact_information DONE
    party_identifier DONE
  • fully-featured-lookup (1)
    Entity Status
    currency DONE (proof case)
  • read-only-time-series (0 in ores.refdata)

    No ores.refdata entity matched; reference entity market_observation lives in another component.

  • workspace-scoped-lookup (8)
    Entity Status
    cds_convention DONE
    deposit_convention DONE
    fra_convention DONE
    ibor_index_convention DONE
    ois_convention DONE
    overnight_index_convention DONE
    swap_convention DONE
    zero_convention DONE

No-profile-match — none remain

Every ores.codegen.entity and ores.codegen.junction file in ores.refdata matches a profile (7/7, after adding tenant-scoped-junction). 100% coverage is achievable without inventing or forcing any additional profile.

Notes

  • business_unit (matched fk-scoped-child) and leg_type (matched simple-lookup) both left a feature unset that their matched profile fixes to a value (has_workspace_id for business_unit, has_change_reason_cache for leg_type) — the survey only compared features already present on both sides, so an unset feature the profile would newly inject (via its setdefault semantics) never showed up as a mismatch. A trial rebind of each surfaced a real, non-empty diff — caught by the regen+diff check exactly as designed, not a tooling bug. Lesson for the next component: a correct match requires comparing every profile-assigned feature against the entity's actual resolved value (defaulted or not), not only features the entity happens to already set. The two cases resolved differently on inspection, not by the same rule:
    • leg_type missing change-reason tracking was a genuine gap, not a deliberate design choice — every other simple-lookup entity audits who changed a row and why; this one arbitrarily didn't. Bound to the profile, which added a ChangeReasonCache* constructor parameter to LegTypeController (same shape every sibling controller already has); wired the one hand-written call site (RefdataPlugin::setupControllers) to pass ctx_.change_reason_cache alongside them.
    • business_unit is genuinely tenant-only, not workspace-scoped — unlike book=/=portfolio (the other two fk-scoped-child members, both truly workspace-scoped), nothing about business_unit's domain semantics calls for per-workspace isolation, and forcing it in would add a real, unwanted workspace_id column (SQL schema + index + validation trigger + C++ struct field) — a schema migration for no product reason. Left unbound, keeping its own explicit properties: this is the entity's genuinely bespoke, correct configuration, not a defect to fix or a profile to force. Not a candidate for a new profile either — the catalogue's own bar is a combination recurring across more than one entity, and business_unit is currently the only ores.refdata entity with this exact shape.
  • All 63 remaining rebinds (55 domain_entity + 8 junction) were applied via a scripted, per-entity pipeline: for each entity, regenerate the unmodified file first to capture any pre-existing drift baseline, revert, apply the profile rebind, regenerate again, and require the new diff to be an exact subset of that baseline (zero additional diff) before keeping the edit — otherwise revert and flag for manual review. This is the same method proven by hand on currency and currency_calendar, just automated across the worklist so every entity still gets its own real regen+diff verification rather than a batch assumption.
  • Rebinding a junction (currency_calendar) surfaced pre-existing, unrelated generation drift — the checked-in currency_calendar_generator.cpp=/=currency_calendar_entity.cpp no longer byte-match what the current templates produce even for the unmodified model file (a missing #include <faker-cxx/faker.h> and a blank-line formatting difference), and several facets (messaging handler/registrar, service) that the current templates emit at --address ores were never generated/checked in for that entity at all. Isolated by regenerating the untouched original file before touching it — confirmed the profile rebind itself introduces zero additional diff. Out of scope for this task (not caused by profile binding); flagged for a separate drift-remediation pass once the full worklist above is regenerated, since the same pre-existing-drift check should be run for every entity in this worklist before trusting its diff as a true zero.

Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario, run through the QA Validation Runner panel) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

PRs

PR Title
#1808 [refdata,codegen] Bind ores.refdata entities to profiles; verify zero-diff regen

Review

# Comment summary File Decision Notes
1 Duplicate/contradictory ignore entry (flagged independently by all 3 review passes) projects/ores.sql/utility/validation_ignore.txt Accepted Fixed in 6f44cc3b4 – removed the stale "intentionally omit version" entry, kept the correctly-explained one.
2 CI has no check that catches ores.refdata drift itself (user request, raised mid-round, not a bot finding) .github/workflows/pr.yml, projects/ores.codegen/scripts/check_component_drift.py (new) Accepted Added in 4a1bba138: refdata-codegen-drift job regenerates every ores.refdata model on every PR and fails on any diff. Scoped to refdata for now; extend –components per component as each is brought to a verified zero-diff state.
3 Minor: legacy is_domain_entity_model fallback path noted, not a defect in this diff projects/ores.codegen/src/codegen/core.py Declined Not changed. Explicitly flagged as no-action-needed by the reviewer itself – a future cleanup note, not a defect in this PR.
4 Minor: silent last-write-wins on impls_lower collision if two implementations UUIDs differ only by case projects/ores.codegen/src/codegen/core.py Declined Not changed. Can't happen today (UUIDs are canonically case-insensitive identifiers, so two such keys really are the same paste point) – noted by the reviewer as an implicit assumption, not a bug.

Result

Delivered, and considerably wider in scope than the original pilot goal, after the user escalated this from "prove the profile catalogue" to a full zero-drift hotfix:

  • :profile: binding implemented for both domain_entity and junction metatypes; 7 profiles (added tenant-scoped-junction for the 9 junctions); 100% of ores.refdata's 67 codegen-eligible entities/junctions bound and zero-diff regen verified, per Acceptance.
  • currency (reference fully-featured-lookup entity) and currency_calendar (reference junction) both rebound and zero-diff-verified, as required.
  • leg_type bound to simple-lookup after a genuine gap (missing change-reason tracking) was found by the profile-injection check; business_unit correctly left unbound (genuinely bespoke, not workspace-scoped) – see * Notes for the reasoning on both.
  • Beyond the pilot's original scope, at the user's explicit direction to get the branch fully green rather than merely proving the tooling: brought party's previously-BLOCKED SQL layer fully under codegen via two new template paste points (plus reuse of an existing one), eliminating the last hand-written/generated divergence in the component; fixed a real, previously-undetected bug in the codegen paste-substitution mechanism (case-sensitive <<paste:UUID>> matching – found only by actually running compass db recreate -y, not by isolated diffing); fixed a composite-component scaffold bug (aggregator components like ores.refdata itself matching the flat single-module archetype); fixed a wire_codec template bug in the NATS event-cache generator; eliminated all remaining ores.refdata generation drift (~130 regenerated files, byte-identical to current templates); silenced one pre-existing, unrelated TEMPORAL_004 schema-validation warning on ores_compute_app_version_platforms_tbl with a documented follow-up note (explicit user instruction – that entity's own regen/filename-migration is out of scope here).
  • Rebased onto latest main (picked up an unrelated ores.ore.core XML fix) and re-verified end-to-end: full clean rebuild, DB recreate, full ctest run. Two failures surfaced post-rebase, both root-caused and fixed (not worked around) rather than dismissed as flaky, per the user's standing "fix it properly" directive:
    • ores.qt.headless.tests (instrument_parse_dispatch_tests.cpp, 10 cases): test fixtures were json-encoded but parse_trade_instrument() decodes via the process-wide default_wire_codec(), which defaults to msgpack absent a real NATS client. Pinned the test binary's default codec to json.
    • ores.iam.core.tests (mint_token_respects_the_requested_ttl=/impersonation tests): a real race, not flakiness -- =mint_token() stamped the token's issued_at=/=expires_at before a DB round-trip (get_effective_permissions()), so a short ttl could expire before the token was even returned under load. Moved ttl stamping to immediately before signing; verified with 5 consecutive stress-test ctest runs, all green.
  • Final state: full local build clean, validate_schemas.sh green, full ctest suite 74/74 passed.

Acceptance criteria (see * Acceptance) are all met. Deferred, by explicit user instruction or genuine out-of-scope: a full regen/filename-migration for ores_compute_app_version_platforms_tbl (follow-up noted in validation_ignore.txt); a same-treatment audit of counterparty against what was found for party (raised by the user mid-session, not picked up this task – worth a fast follow-up given the structural similarity).

Emacs 29.3 (Org mode 9.6.15)