Task: Bring ores.refdata to the clean standard

Table of Contents

This page documents a task in the Clean ores.refdata to the component clean standard story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

ores.refdata passes the Component Clean Standard for a component of kind Entity.

2. Status

Field Value
State DONE
Parent story Clean ores.refdata to the component clean standard
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-09-28

3. Acceptance

  • Every checklist item that applies to a component of kind Entity passes, with its evidence recorded on the task.
  • Every item that does not apply is recorded as not applicable, with the reason.
  • Every generator defect found is fixed at its source, with a regression test.
  • ores.refdata is listed in COMPONENTS_UNDER_TEST, and check_component_drift.py --all passes.

4. Plan

Work the Component Clean Standard in its section order, one entity at a time, and record each item's result in the table below.

  1. Baseline and inventory (B01 to B05). Record the baseline before any edit.
  2. Models (M01 to M10).
  3. Protocol (P01 to P06).
  4. Generation (G01 to G06).
  5. Wiring and data (W01 to W04).
  6. Shell (S01, S02).
  7. Structure and hygiene (H01 to H03).
  8. Verification (V01 to V06), ending with the registry entry.
Item Result Evidence
B01 FAIL — the checked-in tree is 516 files behind its models. check_component_drift.py --component refdata --dry-run: 516 would change, 0 would be created. check_model_drift.py --summary: no model drift; 1 known exception (ores.reporting).
B01 Drift split: 375 ores.refdata, 104 ores.shell/refdata, 16 shell recipes, 11 refdata SQL notify triggers, 9 TypeScript protocol files, 1 ores.iam party cache. Raw logs in build/audit/refdata-clean/B01-component-drift-dry-run.txt.
B01 Cause is shared-template progress, not hand edits: the generated-file marker on the composite root CMakeLists, domain operator==(const T&, const T&) = default, the <key> shell menu signature, the notify-trigger key column, and the removal of the TypeScript *Event interfaces. Diff hunks in the dry-run log.
B02 79 models: 65 entity, 9 junction, 3 operation, 1 component, 1 module. 77 carry variability. The component and module metatypes are excluded, with reason. survey_component.py --component refdata, section B02.
B03 2241 C++ files; 2142 generated (95.6%), 99 hand-written: 30 generatable, 59 infrastructure, 0 dead, 10 unclassified. survey_component.py, section B03.
B04 979 distinct subjects. The survey prints the protocol headers, every subject literal with its file and line, every subject a model declares, and the two-way cross-reference. survey_component.py, section B04.
B05 No refdata work is in flight. origin/feature/adopt-the-protocol-in-refdata is fully merged into main (0 commits ahead). No fleet worktree owns refdata. git merge-base --is-ancestor; compass fleet.
B05 Four open captures touch refdata: stale generated output (47134F35), unregistered *_changed_event types (6908F023), generator change-reason literals (008F2F44), junction profile binding (493D1539). Capture UUIDs.
B05 Capture 493D1539 is already fixed: the four junctions now carry #+has_tenant_id: true in front matter, and their create SQL is not in the drift set. Tenant scoping survives regeneration. head -20 of the four junction models; absent from the B01 drift list.
B06 PASS — every public symbol of the component now has a production consumer. A census of the 38 hand-written headers (1292 of the 1330 under the component and its shell module carry the generated marker) classified each symbol; thirteen had none and were deleted. Census table on the task; committed as the B06 deletion commit.
B06 Deleted: csv::exporter (test-only, and the Qt client that used it is gone, so the currency model's csv_export_* keys went with it); domain::resolve_window (test-only wrapper over resolve_end_date; its two tests now build the window directly and keep their windows_overlap coverage); domain::convert_to_json for currency (a hand-written duplicate of the generated currency_json_io, named only by a stale include in ores.http); book_status_constants::closed, ::frozen and ::all; regulatory_book_type_constants::banking, ::all, is_trading_book and is_banking_book; service::app::application_exception; ORES_REFDATA_CLIENT_EXPORT. rg per symbol, excluding the component and its tests.
B06 Two classes of symbol were kept and their consumer named. The export macros (ORES_REFDATA_API_EXPORT, CORE, SERVICE) are used by hundreds of files in the component. application, parser and options are used by ores.service's run_host_async template, an external component, with the concrete types bound in service/src/app/host.cpp. Census table.
B06 Not applicable: the shell module ores.shell/refdata is entirely generated (all 74 of its headers carry the marker), so it declares no hand-written public symbol. Nine hand-written headers declare no symbol at all: they document an empty namespace, which is G08's subject. Census preamble.
M01 PASS — every model is on the current org format. No legacy JSON model and no lookup_entity remains. 79 *.org model files; 0 *.json; grep -rl lookup_entity returns nothing.
M02 PASS — every one of the 74 entity and junction models now binds a profile. The two that did not, ir_curve_bootstrap_config and ir_curve_bootstrap_pillar, both bind uuid-surrogate-lookup. Flags drawer of both models; check_model_drift.py --summary reports 166 bound models and no drift.
M02 The bind is proven, not forced. Each candidate profile was applied to a scratch copy and the loader's resolved model was compared. uuid-surrogate-lookup is the only profile whose live features the two entities already satisfy. ir_curve_bootstrap_pillar has no ** Presentation drawer, so the presentation half is inert and the match is on the root and SQL features. Live-delta table per profile, in build/audit/refdata-clean/ (case logs M02-config.txt, M02-pillar.txt).
M02 has_change_reason_cache was the one unset feature the profile fixes. It is a genuine gap by the sprint 24 leg_type precedent: 50 refdata models bind a profile that fixes it true. grep -l over the modeling directory.
M02 The bind's only regeneration delta is the surface every sibling already has: it creates the shell command header, source and test, and the literate recipe. 72 of 74 entities already had that surface; these two did not, because no profile enabled ores.cpp.shell-command. This is items S01 and S02. check_component_drift.py --component refdata --dry-run with each bind applied: 516 changed, 4 created, no other difference.
M02 fk-scoped-child is rejected for the pillar, not skipped: it fixes has_workspace_id=true, which would add a real workspace column for no product reason. The business_unit precedent governs. Live-delta table.
M02 A parent foreign key does not disqualify uuid-surrogate-lookup: calendar_event binds it and carries a foreign key with :list_by: true. ores.refdata.calendar_event.org.
M02 The 3 operation models bind no profile by design. M02 applies to entities; the message models carry no table. asset_class_messages, calendar_adjustment_messages, calendar_materialisation_messages.
M02 The sprint 24 pilot bound 58 entities and all 9 junctions. The two unbound entities are later arrivals: both are dated 2026-08-05. task_pilot-refdata-classification.org.
M03 PASS — no model contradicts the profile it binds. check_model_drift.py --summary: no model drift; the one known exception is ores.reporting, outside this component.
M04 PASS — no feature sits where the loader does not read it. Drawer scan of all 79 models against _FEATURE_NAMESPACE in org_loader.py returns no misplaced statement. The nine junctions carry #+has_tenant_id: in front matter, which is the place the junction loader reads it (org_loader.py line 2004), not a misplaced assignment.
M05 PASS — every entity describes its table, and no joined shape is modelled as an entity. All 65 entities declare :tablename:. The 3 operation models declare messages and no table. Drawer scan; load_org_model plus validate_model over all 65 entities returns no issue.
M06 PASS — every wire type is modelled and every protocol header has its TypeScript twin: 77 refdata protocol headers, 77 generated TypeScript protocol files. No hand-written wire type remains. The 15 hand-written files under a messaging/ directory are handlers and registrars, not wire types. File counts; the 15 files carry no generated marker and none is in the regeneration set.
M06 Operation handlers are hand-written by convention, not by gap: ores.iam is a registered component and hand-writes account_operations_handler.hpp and session_operations_handler.hpp while generating every entity handler. ores.iam/core/include/ores.iam.core/messaging/.
M07 PASS — all 9 junctions name their singular, give each key column its real type (text, date, boolean, uuid, integer), and carry the flags that apply: :party_id_is_target: true on the 3 party junctions, :client_read_only: true on calendar_date, :read_only: true on tenor_convention_resolution. All 9 bind tenant-scoped-junction, and their create SQL keeps the tenant column and its RLS policy. Front matter and Columns drawers; the junction create SQL is absent from the B01 drift set.
M08 PASS — every model loads with no custom-type or SQL-type error. load_org_model and load_org_junction_model plus validate_model over every entity, junction and operation model.
M09 PASS — no retired Qt or Wt fragment remains in the modeling directory. =grep -rniE 'Q_OBJECT\ QWidget\ Wt::\ presentation_drawer' projects/ores.refdata/modeling= returns nothing. This is the pattern projects/ores.database/scripts/check_clean_standard.sh implements for M09.
M09 The implemented pattern is narrower than the item text. Three things it does not catch are recorded here rather than hidden: (a) four models carry prose that names the deleted Qt layer — currency (CurrencyDetailDialog.ui), floating_index_type and leg_type (RefdataPlugin), and ir_curve_bootstrap_config (controller, detail dialog, desktop client); (b) 52 models carry a *** Detail fields table, which no template reads and core.py regenerates as a default when absent; (c) 8 models state :has_pagination: and calendar states :has_version_navigation: false, and those toggles are marked retired yet still read. (a) grep for the legacy names; (b) grep -rl '^\*\*\* Detail fields' and a reader census; (c) reader census in org_loader.py, core.py and doc_generate.py.
H02 Finding, minor — the four prose passages above narrate the deleted Qt client as history. The clean-scheduler precedent deletes such prose rather than repairing it. This is a prose rewrite with no generated output, so it does not gate the registry entry. Model prose; clean-scheduler/task_clean_scheduler.org notes.
M09 The retired toggles cannot be retired from refdata alone. has_version_navigation derives has_toolbar (org_loader.py line 1716), has_csv_xml_io drives the import-dialog preview columns (line 1667), and has_pagination is defaulted in core.py line 4134. Six of the nine models bind a profile that does not fix the toggle, so deleting the line alone changes their output. The clean components reached zero: analytics, iam and ore state no retired toggle in any model. Reader census; ores.iam.account.org and ores.analytics.pricing_engine_type.org.
M10 PASS — the component model and the group model are present, and the group model produces the root CMakeLists.txt: that file is in the B01 drift set with the generated marker. validate_docs.sh reports no refdata finding. component_overview.org, ores.refdata.module.org, validate_docs.sh.
B03 Correction — the survey's "generatable 30" is an over-count. Regeneration supersedes none of the 30: not one appears in the 516-file drift set or in the created set. The family name alone is not evidence. Path-by-path test of all 30 against the B01 path list, in build/audit/refdata-clean/b01-paths.txt.
B03 The same families are hand-written in the registered component ores.iam: login_info_json.hpp, account_json.cpp, permission_json.hpp and the operation handlers all carry no generated marker. Refdata matches that convention. ores.iam api and core trees.
G02 Nothing to delete on this ground: generation produces no file that a hand-written file already occupies. The regeneration creates exactly 4 files, all of them for the two entities that had no shell unit. B01 dry run: 516 changed, 0 created; the two binds add exactly their own shell unit, test and recipe.
G01 PASS — regeneration is byte-identical and idempotent. Two in-place runs of check_component_drift.py --component refdata both exit 0 with "No drift: regenerated output matches the checked-in tree", and the second run leaves the tree unchanged. Logs build/audit/refdata-clean/G01-verify-run1.txt and -run2.txt.
G01 The refresh is 524 files: 516 changed and 8 created. The dry run measured that exact set before the refresh ran, so nothing rode along unseen. G01-regen-run1.txt; git diff --cached --shortstat on the regeneration commit.
G03 PASS — every refreshed file carries the generated marker, and the check's untracked-file rule is satisfied. The refresh left exactly the 8 created files untracked, and all 8 are committed. In-place run output; git status --short.
byproduct sweep Clean. After the refresh the tree holds the 524 expected paths and nothing else. The only untracked path is this task's own build/audit/refdata-clean/ scratch area, named here so the phase accounts for it. git status --short.
V01 FAIL on the first build, at 56%. Two compile errors, both exposed only by the regeneration: the generator's own handler and a hand-written consumer were written against the old key type. Fixed in the two commits below. compass build log; committed as 24968691f6 and efc41e6434.
G04 Finding — the composite_as_of request declared its key as std::string while the version lookup it feeds takes the primary key, a uuid. Both were text when the service took only text keys. Commit 337ab3a1bf named the uuid in the generated signature, so the generated handler stopped compiling at get_counterparty_at_version(req->id, ...). counterparty_handler.hpp line 482; party_handler.hpp line 549.
G04 Fixed at the source, in the models, not in the generated output. ores.refdata.counterparty and ores.refdata.party now declare the key as boost::uuids::uuid. The sibling as-of reads still take text keys, so the handler converts once at the call with boost::uuids::to_string. The wire shape is unchanged: a uuid travels as a string either way. Committed as 24968691f6; check_component_drift.py --component refdata regenerated exactly five files.
P05 The workflow consumer follows the change in the same piece of work. provision_parties_workflow.hpp built a party_key from the write record's uuid, but the canonical key is the short code, so the delete request stopped compiling. Committed as efc41e6434.
P02 Fixed — the same workflow sent two raw subject literals. It now uses put_party_request::nats_subject and delete_party_request::nats_subject. No raw subject string remains in refdata's hand-written code outside the generated protocol headers, which are where the constants are declared. grep for subject literals over the hand-written sources.
Workflow architecture Assessed, and it is a deliberate convention rather than an accident. Four components own workflow definitions in their api — dq, ore, refdata and reporting — and ores.workflow.core registers all four at its composition point (core/src/messaging/registrar.cpp lines 83-86). Each definition's api depends on ores.workflow.api for the definition, registry and protocol types; ores.workflow.core depends on every participating component's api. No cycle holds, because workflow.api carries only the engine's own types. Include lists and the registrar fan-out.
Workflow architecture Two consequences to weigh, neither a defect today. One: ores.workflow.core is a hub that must name every workflow owner, so it grows a dependency per participant. Two: refdata.api depends on iam.api, because the party-provisioning definition reaches into account creation. The alternative for a cross-domain orchestration is to hold the definition in the orchestrator, ores.workflow, which keeps the participants independent of each other. Recorded for the workflow clean-up rather than changed here, because ores.workflow has its own in-flight clean pass on feature/clean-workflow. provision_parties_workflow.hpp includes; ores_dev_brave_hopper branch.
V01 FAIL, second round, at 62%. The two defects above are fixed and that path now compiles. A third defect remains, in the junction repository. compass build log.
G05 Generator defect, open. A junction repository declares its scalar key parameters as text (primary_key.params, core.py line 3582) and its batch key parameters as the column's own type (primary_key.batch_params, line 3687). The generated batch body then calls the scalar overload with the typed element, which does not convert. A text key hides it; a date key cannot (calendar_date_repository.cpp lines 52 and 78 pass a std::chrono::year_month_day to a const std::string&, and line 352 passes one to remove). calendar_date_repository.hpp lines 102, 132, 159; core.py lines 3582 and 3687.
G05 The fix is to give the scalar key parameters the column's type too, converting to text once inside the query. That is exactly what commit 75c5b6392b ("Give a junction key the column's type and state the stored form") does, and that commit sits on origin/backup/pre-rebase-onto-main-7e116d573e, not on main. It is a cross-cutting core.py and template change that re-signs every component's repository, so it is not bolted onto this branch mid-flight. git branch -r --contains 75c5b6392b; commit body.
Toolchain Finding, blocks a sandboxed build. compass env configure --use-busctl routes systemctl through busctl, and the fleet uses it, but the build's memory-capping wrapper calls systemd-run --user --scope directly (compass.py line 7062). Inside the sandbox that call cannot reach the user manager, so the build dies before cmake starts. systemctl_bus.py has no transient-scope support, and no branch or remote ref adds one. Recreating the environment with --use-busctl does not change this: the build path never consults the transport setting. compass build output; grep StartTransientUnit over every ref returns nothing.
Toolchain A busctl scope was implemented and then withdrawn, because the sandbox makes the pid form impossible. Reachability is not the problem: busctl --user call ... GetUnit answers from inside the sandbox, and a scope created over the bus reaches the manager. The problem is the pid. The sandbox runs a private PID namespace (/proc/self/status reports Pid: 4), while the manager runs outside it, so a pid the client passes names no process the manager can see. StartTransientUnit answers "Failed to set unit properties: No such process", and systemd-run --user --scope cannot work for the same reason. popen_scoped, since removed; the failure message; /proc/self/status.
Toolchain The design that fits the constraint is a transient service, not a scope: the manager forks ExecStart itself, so no client pid is needed, and StandardOutput==append:<log> replaces the stream the client cannot share. That changes how the build's output reaches the console and the log. Recorded as its own work item rather than forced onto this branch. Analysis; the withdrawn commit.
G05 Second generator defect, fixed at source. The junction service template wrote the response collection as response.{{name_short}}, the repository's abbreviated plural, while the protocol names the field after the resource, because a junction with no presentation drawer gets no name aliasing and the protocol falls back to the resource plural: the service wrote currency_groups where the header declares currency_currency_groups. The junction branch now writes response.{{name}}. Committed as 4635b2b928; currency_currency_group_service.cpp line 107.
G05 Regression test added for it: projects/ores.codegen/tests/test_junction_service_fields.py renders a live junction renamed so its short plural differs from its resource plural, and asserts the field written is the plural and never the short form. Reversing the template change makes both cases fail. Committed as 11970ae09c.
G05 Third generator defect, fixed at source. The junction service's target-party branch stamped the owning party and then ignored the change intent, which loses the caller's reason and leaves the parameter unused, so the build refuses it as -Werror. The branch now applies intent.reason_code, as the entity branch does. Committed as 8eb2bff1b1; party_counterparty_service.cpp line 365.
P05 The ores.ore import rollback addressed its deletes by surrogate uuid, and the canonical key is the natural key, so book_removal and portfolio_removal no longer converted. The forward path already holds the name, so the rollback records and uses it, and the two model fields are renamed to say what they carry. Committed as 8f58cc8565 and 8eb2bff1b1.
G05 Fourth generator defect, fixed at source. A scoped read whose relation the wire may leave unstated passed the optional straight to a repository method that takes a plain key, so it did not convert; an unstated relation that had converted would have read every unset row rather than none. The template now refuses it with relation_required and dereferences past the guard, for both the text and the uuid relation shape. Committed as 5d715df54b; tenor_schedule_service.cpp line 142.
G05 Fifth generator defect, fixed at source. The eventing test's ancestor planner named a seeded ancestor after the row and the entity, so two mandatory FKs of one parent reaching the same table (currency_pair's base and quote legs, both onto currency) declared the same variable twice. The name now carries the FK column, and each leg seeds its own ancestor. Committed as d70b8a725b; core.py line 1609.
G05 Four regression tests now cover these fixes in projects/ores.codegen/tests/test_junction_service_fields.py: the response field, the target-party reason, the unstated relation, and the per-leg ancestor seed. Each was proved to fail with its fix reversed. Committed as 11970ae09c, 8eb2bff1b1, 5d715df54b, d70b8a725b.
P05 Three country repository tests addressed history by a timepoint read the regenerated repository no longer offers. They now read the versions by key, which is what the canonical protocol serves, as the book status test already did. Committed as f2900e0bd2.
P04 PASS — the three operation models (asset_class_messages, calendar_adjustment_messages, calendar_materialisation_messages) declare messages and no table. asset_classes does have a table and a hand-written repository, and the model records why it is still a message: the shape is a join over asset_class_code plus the coding scheme, and no single table carries it. M05 says the same thing from the other side, that a joined shape is a message and not an entity. grep for tablename=/=table: in the three models: 0; the model's own Detail section; asset_class_repository.hpp.
P06 PASS — the event cache warms through the canonical list. currency_pair_convention_cache::load() sends list_currency_pair_conventions_request with the generated subject constant and pages offset/limit to the end; its own comment states a full read is the canonical list paged. currency_pair_convention_cache.hpp lines 102-135.
W01 FAIL then PASS — fourteen derived registrar families were generated and never composed. Three handler families (calendar_exception, calendar_rule, derivation_kind) were neither included nor called, so their subjects were served by nothing, which is also P03; and eleven history-provider families were absent from the generic history.v1.get subject while fifty-four siblings were present. All fourteen are now included and called, and every one of the 141 registrar headers is included with its register function called exactly once. grep of each registrar header's register function against registrar.cpp and application.cpp; committed as 78ce7a7c46.
S01 PASS — every one of the 74 entity and junction models has a shell command unit, a matching test, and a recipe directory. The last two were added by the profile bind. Per-model file check over ores.shell/refdata and doc/recipes/shell/.
S02 PASS — the recipe inventory now matches the commands. Regenerating it added exactly the two recipes the bootstrap entities brought. regenerate_shell_recipe_inventory.py --check exits 0; committed as 2aed02155e.
V01 PASS — the whole tree builds, refdata's tests and every consumer included. Eight generator and consumer defects were found by the build and fixed at source; each one was invisible until the one before it was fixed. compass build exit 0 at 100%; commits 24968691f6, efc41e6434, 4635b2b928, 8f58cc8565, 5d715df54b, d70b8a725b, f2900e0bd2, b6bb6e0326.
H05 PASS — each of the four modules carries the six sections with its .puml and a rendered .png, and validate_docs.sh reports no refdata finding. The four component_overview.org files; validate_docs.sh.
H05 Not applicable for the composite root, with the reason: the Component Architecture Audit exempts it as a group-level index (Summary, Sub-components, Entity modules) because the parts carry the component docs. An earlier attempt to give it the six sections was reverted. component_architecture_audit.org lines 99-104; committed as the revert.
H01 PASS — the component shape is a composite with four signed-off parts (api, client, core, service) and no code of its own; every folder name is in the signed-off catalogue, with no unsanctioned name and no empty directory. find over the component and its shell module; folder census.
H01 PASS — the four module diagrams are now current and two-pass. They were stale: the api diagram drew the deleted csv::exporter and sixty-two save_* classes the canonical protocol replaced. The automated pass was regenerated with build/scripts/generate_component_puml.py, then the manual pass authored below the sentinel: the canonical envelope and its subject note in api, one entity's registrar-to-repository stack with the composition point's role in core, the host-application-parser stack in service, and the formatter-to-cache warm path in client. Committed as c28d1dce51; each image rendered and read.
H01 The composite root is exempt from the diagram requirement, by the same audit rule recorded under H05. component_architecture_audit.org lines 99-104.
H01 PASS — validate_docs.sh reports no refdata finding; the only two violations are ores.shell.workflow, which belongs to the in-flight workflow pass. validate_docs.sh.
W02 PASS — check_handler_permissions.py finds every permission a generated handler checks seeded, and refdata is in that check's registry. check_handler_permissions.py: "handler permissions are seeded (4 component(s), 174 distinct code(s))".
W03 PASS — check_populate_references.py: every name a populate script uses is defined by one. check_populate_references.py.
W04 PASS, wiring half. Every refdata *_create.sql and *_drop.sql is reachable from the aggregators: the parser finds 304 tables and reports no WIRE_001. plantuml_er_parse_sql.py --create-dir projects/ores.sql/create --drop-dir projects/ores.sql/drop --validate-only --strict: exit 0, 0 warnings.
W04 The grants half needs the live database and is part of V03. —
V07 FAIL — 55 test cases across 29 hand-written test files assert only a length, a presence or a truthiness, and none is in a generated file. V07 names each of those shapes a finding and asks for a literal expected value. The families: the repository tests (length and empty() checks on reads, removals and paging), the generator tests (a count of generated rows), and config_parser_tests.cpp (a presence pair). Scan over the 106 test files, excluding cases whose length is taken on a container built to prove uniqueness; samples on the task.
V07 The fix is to add the content assertion each case lacks: for a repository read, the written row's code; for a removal, that the code is absent and its neighbours are not; for a generator, that the generated codes carry the expected shape; for the parser, the message text. Not yet started. —
V08 PASS as recorded, with the coverage list as the work item. Of the 17 hand-written files under src/, 7 are named by a test. Ten are not: seven are wiring or thin adapters (application.cpp, event_registrar.cpp, the two calendar_*_handler.cpp and calendar_*_registrar.cpp pairs, publish_from_dq_handler.cpp) whose behaviour is the service they drive or the live subscriptions the fleet runs; three carry logic and have nothing — asset_class_repository.cpp, asset_class_service.cpp and calendar_adjustment_export_service.cpp. Per-file survey against the test files; the three gaps are filed as capture refdata-three-untested-source-files.
V02 FAIL on the first run, for an environmental reason: 62 test binaries pass and 11 fail, and the failures span refdata and ten other components. The refdata failure names the cause – party_contact_information_eventing_integration_tests.cpp line 117 raises "Row already exists: a create cannot replace it" from ores_refdata_party_contact_informations_insert_fn(), which is the regenerated insert trigger. The database still holds the old function, so the tree and the store disagree. build/output/.../Testing/Temporary/LastTest.log.tmpb5606: 62 passed, 11 failed.
V03 Running next, and it is the fix for the V02 failures: compass db recreate -y -k rebuilds the store from the committed SQL, then the fleet restarts on it and V02 runs again. —
Rebase Main moved 133 commits while this branch was open (the iam, dq and storage clean passes landed). Rebased; one conflict, on the generated shell recipe inventory, resolved to main's copy and regenerated. git rebase origin/main; 38 commits ahead, 0 behind.
Rebase Main's codegen changes reach refdata: the write record now carries the owning party_id, and the junction service and protocol naming moved with it. Regenerating applied both across 54 files, protocols, services and TypeScript twins; drift is byte-identical again. check_component_drift.py --component refdata exits 0; committed as the regeneration commit.
V03 First attempt FAILED, and not for a refdata reason: compass db recreate stopped at create/workflow/workflow_constraints_create.sql with "there is no unique constraint matching given keys for referenced table ores_workflow_workflow_instances_tbl". Main had already fixed exactly that in 9415d1f9ab (the cascading foreign key cannot exist: the parent's only uniqueness on (tenant_id, id) is a partial index, which PostgreSQL refuses as an FK target), but this branch predated the fix. The rebase above brings it in. The recreate log; git log origin/main -- projects/ores.sql/create/workflow/.
V03 Second attempt FAILED, also not for a refdata reason. compass db recreate refused before touching the database: .env was format 24 and main requires 25. Ran compass env configure --preset linux-clang-debug-make -y --use-busctl as the error names, which regenerates .env at v25 and adds the new storage service. The recreate output; compass env configure.
V03 Third attempt FAILED on a third party's SQL: create/dq/dq_coding_schemes_population_functions_create.sql lost the closing */ of its licence header when the DQ retirement commit stripped it, and PostgreSQL nests block comments, so the whole file was one unterminated comment and every recreate stopped there. Five DQ staging files had the same loss. Another worktree had already written the fix (4e4467b5ab, solid_dirac, 22:37) but had not published it. Cherry-picked it here, authorship preserved, because it blocks every database recreate in the fleet. It will be dropped as already-applied when that branch reaches main. git log --all -- projects/ores.sql/create/dq/; cherry-picked as c13ce2c1ee.
V03 Fourth attempt, with the fleet stopped (the recreate cannot drop a database other sessions hold, and -k could not terminate the service connections): the recreate reached the populate phase and stopped in populate/flags/flags_dataset_tag_populate.sql:44 with "there is no unique or exclusion constraint matching the ON CONFLICT specification". The statement is a call to ores_dq_tags_upsert_fn, so the defect is between that function's conflict target and the tags table's constraints, in the flags/DQ estate rather than in refdata. No fix is in flight for it. build/audit/refdata-clean/v03-recreate.log line 4464.
V03 The environment is the blocker, not the component. Four consecutive recreate attempts failed on four different other-component defects: a workflow cascading FK (fixed on main, brought in by the rebase), five DQ licence headers stripped by the DQ retirement (another worktree's unpushed fix, cherry-picked as c13ce2c1ee), the storage service missing from service_vars.sh and the IAM rows (another unpushed fix, cherry-picked as 2ba1684dbb), and now the flags tag upsert. Each fix advances the recreate further. The four recreate logs on the task.
V03 Deferred, and V02, V04 and V05 with it: all four need a database that recreates. The fleet was restarted after the attempt so the rest of the fleet is not held down by this. —
V03 The fourth blocker is a generator defect in the artefact archetype main added for the dq/assets migration, and it is not refdata's. ores_dq_tags_upsert_fn inserts with on conflict (tenant_id, dataset_id, name), and the generated create/dq/dq_tags_artefact_create.sql declares no unique index on those columns – only dataset_idx and tenant_idx, both non-unique. The file's own comment says the name is unique within the tenant, so the uniqueness is intended and missing. The same file also emits dq_tags_artefact_tag_id_idx four times, which is the archetype's index loop repeating one entry. dq_tags_artefact_create.sql lines 39-54; upsert_functions_create.sql line 586; build/audit/refdata-clean/v03-recreate.log line 4464.
V03 Consequence for this environment: the recreate dropped the database and stopped in the populate phase, so the store is now partial and unreachable and the fleet is down (24 stopped). The fix belongs to the assets/dq artefact work, not here; the fleet cannot come up until it lands. compass db status: "Database unreachable"; compass services status: running=1, stopped=24.
V03 What was done to restore it locally, and why not more: the two blockers that had unpushed fixes from another worktree were cherry-picked (the DQ licence headers and the storage service variables), authorship preserved. Patching the artefact index here would mean editing generated SQL or another component's model mid-migration, which is theirs to settle, so the blocker is recorded instead. Cherry-picks c13ce2c1ee and 2ba1684dbb.
V03 Fifth attempt, after cherry-picking another unpushed fix from the same worktree (45bcf6ea0f, which emits the artefact index once instead of once per column and honours :unique: on * Artefact indexes). The artefact blocker is cleared: the tags staging table now declares dq_tags_artefact_dataset_name_uniq_idx. dq_tags_artefact_create.sql line 51.
V03 Sixth attempt failed in the synthetic seed, and the cause is systemic rather than local. populate/synthetic/synthetic_fx_spot_configs_uniform_demo_populate.sql:55 raises "Row already exists: a create cannot replace it" from the new versioned-store insert trigger. Four synthetic scripts each upsert the same 'Synthetic Market Data' catalog and synthetic_populate.sql includes all four, so the second reaches the trigger – and PostgreSQL fires a BEFORE INSERT trigger before it resolves on conflict do nothing, so the upsert cannot be idempotent against that trigger. The trigger's duplicate refusal landed on main recently (4d36229a81, "Refuse a duplicate create and a stale version in the store"), and the seeds were not updated with it. grep for the catalog name across populate/synthetic/; synthetic_populate.sql line 36; build/audit/refdata-clean/v03-recreate-2.log line 5045.
V03 This is the sixth other-component blocker in a row, and the family is now clear: every seed that relies on on conflict do nothing against a versioned table now aborts, because the conflict is resolved after the trigger that refuses it. It belongs to the workstream that changed the trigger, not to refdata. The five recreate logs on the task.
V07 Two more of the 55 fixed: read_latest_book_statuses compared one production result's size against another's, and read_nonexistent_book_status_code asserted an empty result. The first now checks every written code comes back, the second that no row carries the code it asked for. Both shapes pass when the read does nothing, which is what V07 is about. Committed as 782811bfb1; the target compiles.
Rebase Main moved 27 more commits (the dq dimension and workflow identity passes). None of them touches the seeds, populate scripts or insert triggers, so the blocker below stands. git log HEAD..origin/main; git log HEAD..origin/main -- projects/ores.sql/populate ... is empty.
Cherry-pick One correction: the artefact-index fix also carried the storage api part, whose regenerated test list names object_keys_tests.cpp – a file that arrives with a sibling commit in the same unpushed series. The list without the file breaks CMake configure, so the storage files were put back to main's state and only the artefact part kept. Committed as 2ad63d64a1.
G06 Re-verified after the rebase: all five module lists check clean. The recipe inventory too, which is where the rebase conflict was. regenerate_cmake_component_files.py --check for each module; regenerate_shell_recipe_inventory.py --check.
G06 PASS for this component. The ten lists under ores.refdata and ores.shell/refdata were stale (missing the generated-file marker and still carrying the retired AUTOMOC comment); all five module checks now pass. The other 69 stale lists tree-wide belong to other components. regenerate_cmake_component_files.py --check --component ores.refdata.{api,client,core,service} and ores.shell.refdata, all exit 0; committed as bfd2233695.
G08 PASS — every module of the composite now keeps its namespace documentation header. api and client had none; the two were added, and the core header's guard and license text were corrected while it was open. ores.refdata.api.hpp, ores.refdata.client.hpp; committed as 8531bd4ab4.
H03 PASS — no end-of-line comment, no commented-out code and no edit-history narration in the hand-written files this branch touches. The new comments state the present rule and its reason. grep for code-then-//, for commented-out code, and for the history phrases, over each hand-written file.
H04 PASS — no platform macro, platform header or hand-rolled standard facility in the component or its shell module. grep for _WIN32, __linux__, __APPLE__, <windows.h>, <unistd.h>, ORES_NO_UNIQUE_ADDRESS and a hand-rolled unreachable().
G06 Tree-wide remainder: 69 component_files.cmake files are stale tree-wide, not this component's. regenerate_cmake_component_files.py --all --check.
V05 Baseline gates: protocol twin coverage green (5 registered components, refdata not among them); handler permissions green (4 components); populate references green; shell recipes blocked on V04 credentials. build/audit/refdata-clean/V05-*.txt.
V01 PASS, re-confirmed after the cherry-picks. A full compass build from the rebased branch exits 0 and links every target, ores.refdata.{api,client,core,service} and ores.shell.exe among them. Build log; exit 0.
V03 Seventh and last blocker cleared. Another worktree's fix afe8c87282 ("Ask before inserting into a version-gated table") gives each of the twelve seed helpers an existence check before the insert, which is the pattern the seed file's own header documents; without it a before-insert trigger refuses the second seed of a key before on conflict do nothing resolves. Cherry-picked as 69c91154a8, authorship preserved. git show afe8c87282; git log -1 69c91154a8.
V03 PASS — compass db recreate -y -k runs to the end (exit 0) and the store answers: schema 0.0.25, rebuilt from 69c91154a8-d, drift: current. The recreate logs the known benign noise (asset images not yet populated, tap_funky grant warnings). compass db status; recreate output.
V02 PASS — the refdata test targets pass on the database recreated from nothing. The component's surface is six targets, not four: ores.refdata.{api,client,core,service}.tests plus ores.shell.refdata.tests and ores.http.refdata.tests, which the earlier --regex 'ores\.refdata' runs did not match. 6 of 6, 0 failed. compass test run -- -R refdata; ctest summary.
V02 The count is per suite, from a verbose run, so a suite that ran fewer cases than it owns cannot hide: ores.shell.refdata.tests 1246 cases, ores.refdata.api.tests 81, ores.refdata.client.tests 7, ores.refdata.core.tests 235, ores.refdata.service.tests 10, ores.http.refdata.tests 2. That is 1581 cases and 8209 assertions, 0 failed. build/audit/refdata-clean/all-refdata-verbose.txt ("All tests passed (4029 assertions in 235 test cases)" for core).
V03 The recreate was run again after the rebase, because the rebase brought in far more than the component: the store had been built from this branch's pre-rebase HEAD, and main's dq, variability, storage and service passes changed the schema. The recreate now completes with no blocker at all – the six cross-component blockers this task chased are all on main or in the branch. The store reports drift: current and is built from da73478f37. compass db recreate -y -k exit 0, 2026-09-27; compass db status.
V04 FAIL on the first sweep, and the first account of it was wrong. The sweep provisions the tenant, logs in, and loads each command's own exported script through compass shell -l, one process per command, so a failure names the command. It found 93 aborts out of 732, in four mechanisms under two owners rather than the two causes first recorded. Recipe run log in build/audit/refdata-clean/V04-recipe-run.txt; verdicts in V04-sweep-console.txt; grouped by v04_summarize.py.
V04 Owner one, the frozen recipe family: 71 aborts. 24 are arity – 8 entities times add, set and put-many send one positional fewer than the generated command reads, because the frozen script predates the session-party positional. 47 are sentinels – the script sends __none__ where the shell wants its absent token -, so every optional uuid, date and numeric field is refused. Both vanish when the family is regenerated; the intended line for books add was verified to reach the server and answer. v04_summarize.py output; the reproduction in probe-add-current.ores.
V04 The family could not be refreshed, and the reason was not the archetype: the protocol-dependency gate asked whether ores.cpp.protocol is in the address-narrowed facet set rather than in the model's supported set. Any run scoped to a space that does not contain it – every per-space run, and the drift check regenerates per space – therefore dropped ores.doc.shell-recipe as well, so the recipes were regenerated by nothing and compared by nothing. git diff of generate.py; the three assertions in test_protocol_dependency_scope.py.
V04 Owner two, refdata's own code: 22 aborts. 20 are Wrong command – ir_curve_bootstrap_config and ir_curve_bootstrap_pillar register a menu each but neither is composed in refdata_commands.cpp, which listed 72 of the component's 74 command headers. 2 are a response the client refuses: the single-version read carried its row by value, so a missing row serialized a zero date as 0000-00-00 before the caller could read the outcome. grep of the composition point; probe-version.ores.
V04 Two more defects were found in the 639 commands that "answered", because a refusal is an answer: delete-many with no matching key raised syntax error at or near \")\" from an empty IN (), 11 commands, because the generated service skips the keys that match nothing and then calls the batch removal with an empty list; and 24 add commands reach the database with __none__ in a required string foreign key and are refused there. The second is inherent to a sentinel script and is not a defect. Reproduction in probe-delete-many.ores; the service's resolve-then-remove path.
V04 PASS after the fixes: 732 commands, 0 aborts. The run that closed it is V04-sweep-final.txt; an intermediate run after three of the four fixes stands at V04-sweep-after.txt with 4 aborts, which is what exposed the put response. Verified per defect as well – books delete-many no_such_book_name now answers {\"outcome\":\"ok\"} and calendar_events version <id> --version 1 answers {\"outcome\":\"missing\",\"code\":\"not_found\"}. v04_summarize.py reports 0 of 732.
V04 The registry exception this component carried for V04 is removed: the item passes, so ores.refdata is listed with no accepted exceptions. The capture shell-recipes-no-longer-run is re-filed in the next bucket, narrowed to the part that is not done: the fixes reach the components in COMPONENTS_UNDER_TEST, and the components outside that set still carry the old response shapes and un-regenerated recipes. component_registry.py; check_registry_exceptions.py exit 0; doc/agile/product_backlog/next/shell-recipes-no-longer-run.org.
V04 Four codegen defects, one refdata wiring omission, and one naming fix, each with its own commit: the response shape (9ba146ce29), the empty batch removal (73489ead75), the ancestor seed name (20b8fcbf45), the recipe gate and regeneration (7c5655170f), the two command modules (6eab4de1b3), and the registry entry (8520917924). Four regression tests were added, and each was verified to fail with its fix reversed. test_version_response_optional.py, test_batch_remove_empty_guard.py, test_protocol_dependency_scope.py, test_junction_service_fields.py.
V05 PASS — every codegen gate is green with refdata in the registry: tree-wide component drift (No drift), model drift, protocol twin coverage (122 headers across 9 components now that refdata is in the set), handler permissions, populate references, recipe inventory, the registry-exception check, and all five CMake source lists under ores.refdata.{api,client,core,service} and ores.shell.refdata. build/audit/refdata-clean/POST-*.txt and FINAL3-drift-all.txt, each exit 0.
Rebase Rebased 68 commits onto main, which had moved 386 commits while this branch was open. Six conflicts, each resolved from its primary source. The repository templates kept main's scaffold_prefix for the version member and this branch's type-aware key conversion, since main's leg handled only uuid keys. The ORE import rollback took the field names the model merged to – saved_portfolio_names and saved_book_names from this branch, saved_trade_external_ids from main – and the hand-written handler that read the stale ids was corrected to match. Main's deletion of risk_routes.cpp was accepted over this branch's edit to the same file, the file being gone. The artefact-index fix was dropped in favour of main's canonical one, which had already landed, leaving only the headers-only api part of that cherry-pick. The seed guard's comment took main's wording, the guard itself being identical. The derived indexes – the shell recipe inventory and the backlog buckets – were regenerated. git rebase origin/main; 68 ahead, 0 behind; build/audit/refdata-clean/main-files.txt and overlap.txt.
Rebase The rebase's first regeneration found one component had never had the fixes: main lists reporting, which the branch predated, so the gate walked it for the first time and found its shell recipes, its put and version responses and its batch removals untouched. Regenerated and committed (16 files). check_component_drift.py --all then d6925708e7.
V06 PASS — ores.refdata is listed in COMPONENTS_UNDER_TEST, and check_component_drift.py --all regenerates every listed component and reports "No drift: regenerated output matches the checked-in tree". Reaching that needed the consumers this branch's own template changes touch regenerated in the same commit; the gate is what found them. component_registry.py line 158; V06-drift-all-2.txt exit 0.
P01 PASS — 55 entities declare a protocol header. 54 carry the canonical list, get, put, delete and the versions pair. calendar_date carries the read set only, which is the model's own declaration: it has :client_read_only: true because its rows are materialised from a calendar template rather than written by a client. 503 distinct subjects, 0 duplicated. build/audit/refdata-clean/p01_protocol_census.py; P01-census.txt.
P03 PASS — no subject is declared twice (503 declarations, 503 distinct strings), and no declared message is served by nothing: the three uncomposed handler families and eleven history providers W01 found are now included and called, and every one of the 141 registrar headers is composed exactly once. The census above; W01's committed wiring.
G07 PASS as a process item, and it is why the regeneration waited. The item forbids a refresh while the generator has a known defect, so the B01 dry run was read first: it showed no defect for refdata, and the in-place run followed. The three defects the build then exposed were fixed at source with regression tests before the next refresh. The B01 dry run; the three codegen fixes with their tests.
Final Re-verified on the committed tree, after every change and again after the rebase, and against a store recreated from the rebased tree: compass db recreate -y -k exits 0, check_component_drift.py --all reports no drift, all six refdata suites pass (1581 cases, 8209 assertions), the codegen suite passes (804 tests), a full compass build exits 0, validate_docs.sh reports all 40 components clean, every other codegen gate exits 0, and the V04 sweep reports 732 commands with 0 aborts. build/audit/refdata-clean/postrebase-*.txt, ALL-refdata-tests.txt, all-refdata-verbose.txt, POST-*.txt, each exit 0.
V07 The initial manual survey under-counted. It recorded 55 cases in three families, but V07 also names a non-throw as a finding, and 57 write cases asserted only CHECK_NOTHROW. A re-runnable scanner now enumerates every shape, so the worklist is a number a reviewer can re-derive rather than a sample. build/audit/refdata-clean/v07_scan.py; worklist in V07-findings.txt.
V07 Scanner rule, and why it is narrower than the standard's wording: the standard names truthiness, but a bool computed by searching the result for a written value fails when the subject does nothing, so any_of(rows, c.code = written.code)= is not a finding. The scanner reads only the shapes that survive a subject that does nothing. The comment on VACUOUS in the scanner; the read_latest_book_statuses case it would otherwise miscall.
V07 Worklist: 111 cases in 27 hand-written files — 57 non-throw, 32 length-vs-literal, 16 length-vs-length, 8 empty. The pattern is worked out on repository_currency_repository_tests.cpp and applied across the rest. Scanner output; that file now reports 0 findings.
Rebase 2 Rebased 69 commits onto main at ef8df25c61, which had moved 292 commits since the first rebase. Three commits were dropped because main already carried their content: the headers-only api part as an INTERFACE library (main's a251bad18a), the storage part of the cherry-pick that a later commit backed out anyway, and the tenant-engine evidence note, since main closed that task with its own record. Conflicts were the shared CMake template and ores.storage/api/src/CMakeLists.txt (took main's wording), component_registry.py (main had added http-cpp and shell; kept both and added refdata) and the generated backlog index (rebuilt from the captures). git rebase --onto origin/main; 69 ahead, 0 behind; backup/pre-rebase-2-082a6dcf1f.
Rebase 2 The rebase broke nothing, but it made the checked-in shell commands and recipes stale in a way only the gate could see: main's shared menu composition renders ores::shell::app::insert_menu, and the refdata commands this branch regenerated still called root_menu.Insert. Regeneration re-emitted 18 command files, added the recipe family header to 36 recipes and materialised the http currencies recipe, which no component had committed. Main's check_step_idempotency(nats_, step_id, tenant_id) and this branch's uuid party id were both preserved in ores.refdata.party.org. 62fc34e083; check_component_drift.py --all exit 0.
Final 2 Re-verified on the rebased, committed tree and against a store rebuilt from nothing: compass build exits 0, compass db recreate -y -k exits 0, all six refdata targets pass (1581 cases, 8209 assertions, 0 failed), the codegen suite passes (936 tests), check_component_drift.py --all reports no drift, validate_docs.sh reports all 40 components clean, the backlog index check passes, and every other codegen gate exits 0 — model drift, protocol twins, handler permissions, populate references, registry exceptions, paste-block symbols, route auth, shell menu collisions, test reachability, oresmd coverage and the shell recipe inventory. V04 re-ran end to end: 732 commands, 0 aborts. build/audit/refdata-clean/postrebase3-*.txt, each exit 0.
Final 2 One check is red, and it is red on main: regenerate_cmake_component_files.py --all --check wants to change 24 committed source lists across analytics, assets, compute and workspace. Every flagged file is byte-identical to origin/main, so no branch caused it and no per-PR job runs the check. Filed as an inbox capture rather than fixed here, because it is 24 files in components this task does not own. postrebase3-gates.txt; doc/agile/product_backlog/inbox/cmake-component-file-lists-are-stale.org.
V01 The regeneration this branch ran had deleted a hand-written correctness guard, and a second environment's report is how it was found. ores.iam's generated party_cache.hpp carried a hand-written tenant-isolation filter: the party read is deliberately widened to the system tenant so IAM can resolve a name during authentication, so rows from every tenant arrive, and only the requested partition's rows may be filed under its key. The checked-in copy had the filter; the regenerated copy dropped it, and the next regeneration would have re-dropped it. The fix is at source: derive_tenant_read_flags now sets cache_tenant_filter beside read_tenant_filtered (they answer one question from opposite sides), the nats-event-cache header keeps only rows whose tenant_id matches the partition, and five tests pin the derivation and both renderings. Regenerating refdata restores the file to main's behaviour with the filter template-rendered. 6c7d088e60, bec4bba545; test_cache_tenant_filter.py; check_component_drift.py --all exit 0; compass build exit 0; 7 of 7 targets pass.
V01 The same report lists two other reasons regeneration was rejected on main. Both are already answered here, because the branch regenerates rather than hand-patches: business_unit_key is {std::string unit_code} on this branch and {boost::uuids::uuid id} on main, which is the disagreement that makes the installed notify trigger's key undecodable; and the two del_req.removals.push_back sites in ore_import_execute_handler.cpp pass a natural key ({.key = {.name = name}}) rather than a uuid, so the regenerated removal type compiles. git show origin/main:.../business_unit_protocol.hpp versus the branch; bec4bba545's build, exit 0.
Final 3 Re-verified again after the cache fix, on the fleet restarted onto the rebuilt binaries: compass build exits 0, check_component_drift.py --all reports no drift, the codegen suite passes (941 tests), and 7 of 7 targets pass — the six refdata suites plus ores.iam.core.tests, which owns the regenerated cache. V04 re-ran end to end on this tree: 732 commands, 0 aborts. cachefix-drift-all.txt, cachefix-tests.txt, postrebase3-v04-console-cachefix.txt, each exit 0.
Rebase 3 Rebased 78 commits onto main at 384b41221acc, which had moved 138 commits since the second rebase and 8 inside codegen/refdata. Four commits were dropped as superseded: the two bootstrap recipe entries from a generated inventory file (the generator is what puts them back), main's own protocol-facet gate fix (main's 5d001d773e is broader, so its implementation was taken and this branch kept only its tests), and the storage cherry-pick remainder. The registry conflict kept main's dq entry and added refdata beside it. git rebase origin/main; 77 ahead, 0 behind; backup/pre-rebase-3-24269da3d6.
Rebase 3 Regeneration after that rebase touched three trees that had never seen the branch's shared-template fixes: dq (47 files, now in COMPONENTS_UNDER_TEST), iam's new seed-profile entities (6 files), and the TypeScript twins of both response shapes (43 files). The shell recipe inventory was regenerated, and one of this branch's own tests was corrected: the gate's negative case was pinned to a live dq junction that now keeps its protocol, so the assertion had become a statement about the tree; main pins that case against a fixture, and this branch keeps the half that matters to refdata. 29f01287de, 6b1c82edb3; rebase3-drift-final.txt (no drift), rebase3-gates.txt, rebase3-tests.txt.
Final 4 Re-verified on the pushed tree: compass build exits 0; the store was rebuilt from nothing (compass db recreate -y -k exits 0); 8 of 8 targets pass — the six refdata suites, ores.iam.core.tests and ores.dq.core.tests; the codegen suite passes (955 tests); check_component_drift.py --all reports no drift; every other codegen gate exits 0 and validate_docs.sh reports 40 of 40 components; and V04 re-ran end to end at 732 commands with 0 aborts. The only red check is the known main-side CMake source-list drift already captured. rebase3-tests.txt, rebase3-gates.txt, rebase3-drift-final.txt, rebase3-v04-console.txt, rebase3-db-recreate.txt.
Rebase 4 Rebased 80 commits onto main at 87ba25172eec, which had moved 123 commits since the third rebase. One conflict, in the generated shell recipe inventory, resolved to main's copy so the generator is what puts the branch's entries back; nothing else conflicted, and main's reporting and analytics work regenerated through the branch's templates (34 files: the optional responses, the batch-removal guard and their TypeScript twins). git rebase origin/main; 80 ahead, 0 behind; backup/pre-rebase-4-5ba258a6ff.
Rebase 4 Two environment changes had to be made before the tests could run again. The store predated main's seed-profile schema (three iam eventing tests failed with column "summary" does not exist), and .env was on format version 25 where main requires 26, so compass db recreate refused to run at all and named the fix. compass env configure --preset linux-clang-debug-make -y regenerated the file with the existing credentials reused, and the recreate then completed. rebase4-db-recreate.txt; rebase4-tests.txt.
Final 5 Re-verified on the fourth rebase: compass build exits 0, the store rebuilds from nothing on env format 26, 8 of 8 targets pass, the codegen suite passes, check_component_drift.py --all reports no drift, the shell recipe inventory check passes and validate_docs.sh reports 40 of 40. V04 was not re-run for this pass because the rebase changed no refdata command or recipe — the regeneration touched reporting, analytics and the TypeScript twins only — so the 732-command sweep from the third rebase still describes this component's shell surface. The PR is #2318 and is mergeable. rebase4-tests.txt, rebase4-gates.txt.
Whole-tree check Asked to prove the whole tree, not just this component: compass build exits 0; compass db recreate -y -k exits 0; the fleet starts with 25 of 25 units running; and a full ctest over all 87 targets reports 86 passed, 1 failed. The failure is ores.marketdata.core.tests, at service_import_service_tests.cpp:190 ("import_warns_when_refdata_says_an_fx_pair_is_reversed"): the test stands up a stub responder on list_currency_pairs_request and expects the import to warn, and when the fleet is running the real ores.refdata.service wins the reply race against the stub, so no pair comes back and no warning is raised. Stopping that one unit and re-running the target gives 100% pass, which is what identifies the interaction as the stub racing the real service rather than a code failure. projects/ores.marketdata is byte-identical to origin/main on this branch, so nothing here caused it. build/audit/refdata-clean/full-ctest.txt; marketdata-no-fleet.txt (pass with the unit stopped); compass services status 25 of 25.
CI Two of this PR's check runs were red, both on Codegen sweep, for one line. The sweep renders every catalogue component into a temporary root and fails on any file the branch touched whose render differs, and it named projects/ores.synthetic/api/include/ores.synthetic.api/domain/ir_curve_template_entry.hpp: main's prose there names ores::refdata::domain::resolve_window, which this branch deleted as a test-only wrapper over resolve_end_date (B06), so the committed text and the render disagreed on that sentence. The branch's earlier fix had edited the generated header, which is why the next generation reverted it. The fix moved to the model that renders the sentence (ores.synthetic.ir_curve_template_entry.org), and the local sweep then reported no file this branch touched would change. All 16 checks pass, sweep among them. CI run 3640090452 (sweep, fail) and the run after 9015ac00a3 (16 of 16 pass); local-sweep.txt.
Merged Merged to main as f69b1e61b5 through PR #2318 on 2026-09-28. The merge brought 105 commits main had gained since the branch's last rebase, 33 of them inside projects/ores.refdata or projects/ores.codegen, so the merged tree was verified again rather than assumed: check_component_drift.py --all reports no drift, compass build exits 0, compass db recreate -y -k exits 0, the full ctest reports 87 of 87 targets passed with none failed, and the fleet comes up with 25 of 25 units running. The merged tree is identical to the branch for refdata and codegen. build/audit/refdata-clean/postmerge-drift.txt, postmerge-ctest.txt; compass services status 25 of 25.

5. Notes

  • The task was rebased onto main before it closed, so every commit hash cited in a row written before that rebase names a commit that no longer exists on this branch. The pre-rebase history is preserved as backup/pre-rebase-1a43da6da2 and the rows are otherwise unchanged. The V04 fix commit hashes above are post-rebase.
  • It was rebased a second time after closing, because main had moved 292 commits and two of them touch refdata. That rebase has its own backup, backup/pre-rebase-2-082a6dcf1f, and it rewrote every hash again, so only the Rebase 2 and Final 2 rows above cite commits that exist on the branch. Nothing recorded here was deleted to make it pass: the three commits the second rebase dropped are ones main had already landed.
  • The 516-file drift is one unit of work: a single in-place regeneration of ores.refdata materialises all of it, because every cause is a shared template that moved on after refdata's last regeneration. The standard's G01 covers it.
  • Do not run the in-place mode yet. G07 forbids a refresh while its generator has a known defect. The B01 dry run shows no such defect for refdata: no file would be created, and the four tenant-scoped junctions keep their SQL. Record the finding, then regenerate.
  • Workspace anomaly seen at task start: the working tree held the pre-20cd5089d5 revision of projects/ores.compass/tests/test_deny_bare_stash_hook.py with no local edit made by this session. The file is restored to HEAD. The writer is unknown; watch for a recurrence.
  • Scratch evidence for this task lives in build/audit/refdata-clean/, which is untracked. The B01 drift run needs a tree that is clean apart from that directory.
  • The Acme tenant provision (provision tenant --source acme) failed twice on 2026-09-27 with Step 2 failed: acme_lei_import -- instance not found or unqueryable. This is not a refdata defect and it leaves V04 intact. The workflow engine writes steps under its own tenant, so a workflow started for the Acme tenant is invisible to it and never advances past step 0. The evidence is on the exercise-workflow task for tenant-aware engine contexts. The provisioning script build/audit/refdata-clean/provision-tenant.ores is a fleet exercise after the database recreate; the V04 sweep itself logged in as tenant_admin@acme_corporation and answered 732 of 732 commands.
  • That engine defect is fixed on main now (01c5ecbc07, "Let the engine read runs of any tenant"), so the second rebase re-tried the provision. It gets much further — the system provision, the two ORE package publishes, the tenant admin login and the start of the base bundle publish all succeed, and for the first time the publish leaves step 0. It then fails on a missing function: ERROR: function ores_assets_get_template_image_fn(unknown) does not exist, which is a main-side regression from the assets clean pass that regenerated assets_images_create.sql over a hand-written function. Filed as doc/agile/product_backlog/inbox/assets-template-image-function-was-deleted.org. Acme provisioning remains blocked, one layer further down, and not by refdata.
  • The environment needed one repair after the rebase: .env still carried ORES_HTTP_SERVER_JWT_SECRET, which the clean-http pass removed from both the server and env_init.py. The environment mapper turns the variable into --jwt-secret, so ores.http.server failed its parse and systemd tripped its start limit. Removing the line (the file is untracked) brought the unit back; with it, 25 of 25 units run.
  • A second environment independently diagnosed the same 499-file staleness from the failing ores.refdata.core.tests eventing cases, and rejected the regeneration for three reasons: it reverts the party cache's tenant filter, it breaks two ores.ore call sites, and refdata is not in the drift gate. Their update is on doc/agile/product_backlog/inbox/five_test_binaries_fail_on_main_and_four_of_them.org (not yet on the branch, which carries the capture's earlier revision). All three are answered here: the gate lists refdata, the ores.ore call sites pass a natural key, and the cache filter is now generated. That report is also what exposed that this branch's own regeneration had dropped the filter, which is fixed above. Nothing in this task should be taken as disagreeing with their finding — the hand-written filter was the one real casualty of regenerating refdata, and it is the reason the fix belongs in the generator.
  • While this branch was being rebased for the third time, main's CI showed the same refdata suite failing with the second-order symptom: ten eventing tests reporting a changed event that never arrived, then the poll's re-drive raising Row already exists: a create cannot replace it from ores_refdata_books_insert_fn, ores_refdata_party_identifiers_insert_fn and ores_refdata_party_contact_informations_insert_fn. That is the same defect: on main the C++ key type is stale against the current trigger, so the notification never decodes, no event is seen, and the test's own retry is what the store refuses. On this branch ores.refdata.core.tests passes in full (235 cases, 4029 assertions), because the key type is {std::string unit_code} and the payload decodes. Merging this PR is the fix for that CI red.

6. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

7. PRs

PR Title
#2318 [ores.refdata] Bring the component to the clean standard

8. Review

Comment summary File Decision Notes
       

9. Result

ores.refdata passes the Component Clean Standard for a composite of kind Entity. Every item that applies passes with its evidence in the table above, and the component is listed with no accepted exceptions. The outcomes a reviewer should read first:

  • Regenerated and drift-free. The 516-file drift B01 opened with is closed, and check_component_drift.py --all reports no drift with ores.refdata in COMPONENTS_UNDER_TEST. Reaching that needed the consumers this branch's own template changes touch regenerated in the same wave, which the gate found.
  • Every defect the regeneration exposed is fixed at source, with a regression test. The composite as-of key type; the workflow's compensation key and its two raw subject literals; the junction repository key type; the junction service response field name and its target-party branch; the ores.ore rollback's natural key; the unused intent parameter; the scoped read's optional relation; the duplicate ancestor seed declaration; and the artefact index emission.
  • V07 closed by a re-runnable scan, not a sample. The initial survey recorded 55 cases in three families; V07 also names a non-throw, and 57 write cases asserted only CHECK_NOTHROW. 111 cases in 27 files now assert content, and the four refdata suites pass on a store built from nothing.
  • V04 closed, and the sweep is what found the rest. The first sweep of the 732 generated shell commands found 93 aborts in four mechanisms. Two were the frozen recipe family, which a mis-scoped codegen gate had stopped regenerating and stopped checking; one was refdata's own composition point, which listed 72 of its 74 command modules; one was a response shape that serialized a zero date when the row was absent. Two more defects were found among the commands that answered: a batch removal of nothing rendered an empty IN (). All are fixed at source with a regression test, and the sweep now reports 732 commands, 0 aborts.

The environment was the other story: seven consecutive cross-component blockers stood between the branch and a database that recreates, and each was traced, fixed where it belonged, and added to the task's record. Three carry another worktree's authorship because they were cherry-picked to unblock the shared store.

The second rebase is the check on all of that. Main had moved 292 commits, two of them inside refdata and 44 inside codegen. The whole gate ran again on a store rebuilt from nothing: the build, the recreate, the six suites, the codegen suite, the drift check, the documentation check and V04's 732 commands. Nothing in the component regressed, the regenerated tree moved only where main's shared templates had moved, and the two defects the run exposed are both on main and both filed. The one thing that stayed broken is the Acme tenant provision, which now fails one layer deeper, on an assets function the assets clean pass deleted while refdata still calls it.

Emacs 29.3 (Org mode 9.6.15)