Task: Generate data-scope populate archetypes from dataset org-models

Table of Contents

This page documents a task in the Resolve codegen model unification blockers story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

Populate/seed (sql_*_populate, sql_dataset_*, refdata, batch) generate via the unified resolve_targets graph traversal, driven by dataset org-models that declare their supported archetypes in a :PROPERTIES: drawer, replacing the legacy get_template_mappings path and the standalone data JSON files.

2. Status

Field Value
State DONE
Parent story Resolve codegen model unification blockers
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-06-28

3. Acceptance

  • Each seeder dataset is an org model holding its payloads + a drawer enabling specific ores.sql.populate.* archetypes; codegen generates the populate SQL via the graph, byte-identical to current output.
  • ores.sql.populate archetypes are addressed graph nodes, default-off (opt-in); resolve_output_path gains a data branch for dataset/prefix placeholders.
  • get_template_mappings, get_populate_template_mappings and get_non_iso_currency_template_mappings are retired.

4. Plan

4.1. Discovery findings (2026-06-28) — the problem is smaller than feared

  • Only slovaris has a live model. It uses two legacy shapes: the model.json batch manifest (files[] -> {name,model,template}) and the bare-filename map get_template_mappings (country_currency.json etc.). Driving 8 templates: catalog, methodology, dataset, dataset_dependency, tag, flag, currency, country populate (+ a master include).
  • Shape #3 (*_data.json / dataset.code) is DEAD at runtime — zero *_data.json files exist; get_populate_template_mappings / get_non_iso_currency_template_mappings and the is_data_model dispatch branch never fire. The committed fpml/, refdata/, dq/ populate .sql are orphaned artefacts whose models were removed.
  • sql_populate_function_refdata is schema, not data — migrates into ores.sql.schema (prime origin's consolidation area; out of scope here).
  • seeder.sh is broken — it execs a nonexistent projects/ores.codegen/src/generator.py; must be repointed at the graph path.
  • Engine seam (archetype-granularity activation + #+default: disabled) is already built in B8. Facet stub ores.sql.populate.org exists (empty).
  • Heavy data ENRICHMENT lives in core.py (flag SVGs + currency defaults ~1483, dataset dependency expansion ~1526) tied to the JSON files — must be preserved.

4.2. Design decisions

  1. Retire the dead shape #3 rather than migrate it: remove get_populate_template_mappings, get_non_iso_currency_template_mappings, the is_data_model dispatch branch + dataset.code output block, and the dead orphan docs (sql_non_iso_currency_populate, sql_dataset_refdata, sql_populate_refdata). No models -> output-neutral.
  2. Migrate the live slovaris archetypes (8 + master include) into ores.sql.populate as addressed, #+default: disabled nodes; #+output: projects/ores.sql/populate/{dataset}/{prefix}_<x>_populate.sql.
  3. Dataset-as-org-model (per B8 Decision B): promote datasets/slovaris/dataset_overview.org to the model — #+type: ores.codegen.dataset, a drawer enabling its ores.sql.populate.* archetypes, and #+ keys for prefix/dir; payloads referenced (the existing JSON files), not inlined, so the core.py enrichment is reused by a new dataset loader.
  4. Engine: resolve_output_path gains a dataset branch ({dataset},{prefix}); a dataset loader feeds the referenced JSON payloads (+ enrichment) into the template context.
  5. Repoint seeder.sh at the graph (codegen.sh generate --model dataset_overview.org --address ores.sql.populate).
  6. Verify byte-identical against the committed solvaris_* outputs (note the slovaris-dir / solvaris-prefix divergence — standardise as part of this).

4.3. COORDINATION (important)

prime origin is concurrently consolidating the schema create templates (B1+B7+ temporal) on feature/migrate-country-unified-model, heavily editing the SAME generate_from_model dispatch ladder + mapping functions in core.py. To avoid core.py merge conflicts, sequence: do the conflict-free doc-graph + dataset-doc work here now (library/templates + projects/ores.seeder only), and land the core.py engine changes (steps 1,4,5 retirement/loader/branch) AFTER prime origin's schema consolidation merges, then rebase.

5. Notes

5.1. Step 1 — retire dead shape #3 (code) — DONE (2026-06-29)

Removed from core.py (all provably dead — zero *_data.json models exist): get_populate_template_mappings, get_non_iso_currency_template_mappings, is_entity_data_model, the is_data_model variable, the is_data_model dispatch branch, the data return in get_model_type, the dataset.code output-naming branch, and the ~87-line is_data_model enrichment block. Verified: 26 tests pass, module imports, and the live slovaris legacy path still generates all 9 solvaris_* files unchanged (output-neutral).

Deferred to B9/B10 (doc-side, entangled): the 3 now-dead orphan docs + .mustache (sql_non_iso_currency_populate, sql_dataset_refdata, sql_populate_refdata) and their doc-links in currency.org, sql_populate.org, architecture.org. Left as dead artefacts for now to avoid currency.org conflicts with prime origin and because legacy-literate-doc retirement is B10's scope.

5.2. Steps 3-4 progress (2026-06-29)

DONE (committed):

  • Step 4 primitives: ores.codegen.dataset -> model_type dataset; resolve_output_path dataset branch ({dataset}/{prefix}); load_graph captures #+data_source on archetypes. 27 tests.

REMAINING WIRING — approach (B), reuses the existing per-file enrichment (lower risk than a merged-context refactor):

  1. Put #+data_source on each of the 9 populate archetype docs — the dataset- relative payload file the template renders from: catalog->catalogs.json, methodology->manifest.json, dataset->datasets.json, dataset_dependency->datasets.json, tag->tags.json, flag/currency/country->country_currency.json, batch_execute->model.json (the manifest, for {{#model.files}}).
  2. load_org_dataset_model (org_loader): read the promoted dataset doc -> {dataset: {name, prefix}} (for resolve_output_path) + the dataset dir; wire into load_model routing.
  3. resolve_targets: thread each unit's data_source through; read the dataset name/prefix from the model for output resolution.
  4. _generate_single: for dataset units, call the EXISTING generate_from_model(dataset_dir/data_source, target_template, target_output=, prefix=)= — the legacy per-file path — so all enrichment (flag SVGs, currency defaults, dataset-dependency expansion, manifest methodology lift) is reused unchanged.
  5. Promote datasets/slovaris/dataset_overview.org: #+type: ores.codegen.dataset, #+name:/#+prefix: (solvaris), drawer enabling its 9 ores.sql.populate.* archetypes (default-off facet -> opt-in).
  6. Repoint seeder.sh at codegen.sh generate --model dataset_overview.org --address ores.sql.populate.
  7. Verify byte-identical vs committed solvaris_* (standardise the slovaris-dir / solvaris-prefix divergence as part of this).

This is the end-to-end wiring + verification — best done as one focused pass.

5.3. Remaining wiring — DONE (2026-06-29)

Executed approach (B) end to end; the data-scope path now generates through the graph:

  • load_org_dataset_model (org_loader) -> {dataset: {name, prefix}}; routed in load_model on #+type: ores.codegen.dataset.
  • resolve_targets threads each archetype's #+data_source: into its unit; _generate_single renders dataset units from dataset_dir/data_source via the existing per-file generate_from_model (is_processing_batch=True suppresses the model.json batch dispatch), reusing all enrichment.
  • 9 archetype docs carry #+data_source: (catalog->catalogs.json, methodology->manifest.json, dataset/dataset_dependency->datasets.json, tag->tags.json, flag/currency/country->country_currency.json, batch_execute->model.json).
  • dataset_overview.org promoted: #+type: ores.codegen.dataset, #+name:/#+prefix: solvaris, drawer :ores.sql.populate.enabled: true (opts the default-off facet in -> all 9 archetypes enable).
  • seeder.sh repointed at codegen.sh generate --model dataset_overview.org --address ores.sql.populate; generate_solvaris_refdata.sh already redirects here.

STANDARDISATION (full, per decision): the populate tree had two divergent committed dirs — live populate/slovaris/ (current-code output) and stale populate/solvaris/ (May, different code), with the DB actually \ir'ing solvaris/solvaris_populate.sql. Resolved to a single populate/solvaris/: dir==prefix==solvaris; master renamed \{prefix}.sql -> \{prefix}_populate.sql (in the archetype #+output: and the generate_from_model master special case) to match every sibling <dir>/<dir>_populate.sql master and the existing catalogues_populate.sql \ir (no consumer change needed). Deleted the stale slovaris/ dir.

VERIFICATION: the 8 populate files are byte-identical to current-code output; the master is byte-identical content, renamed to solvaris_populate.sql. seeder.sh generate slovaris is idempotent. 30 codegen tests pass (added tests/test_data_scope.py for the loader, output-path branch, and unit threading).

SCHEMA-DRIFT FIX (surfaced by the DB-load verification): standardising to the live seeder output broke compass db recreate — the currency template emitted column asset_class, but the schema had renamed it to monetary_nature (the previously-loaded solvaris/ dir was an older, schema-matching artefact; the live template output had never been validated against the schema). A full static cross-check (every insert column-list vs create table, every upsert call's arg-count vs its signature) found this was the only mismatch across all 8 archetypes. Fixed in ores.sql.populate.currency.org (insert target column + summary query references; the internal VALUES-CTE alias and the {{asset_class}} data lookup are schema-independent and unchanged), re-tangled, regenerated.

DB VERIFICATION: compass db recreate -y -k now completes clean — solvaris populates 100 currencies + 100 countries; monetary_nature carries 4 distinct values. Seed loads end to end.

Remaining: raise the PR.

6. PRs

PR Title
#1369 [codegen] data-scope: generate populate seed via the graph

7. Review

# Comment summary File Decision Notes
1 Stale comment: master include is {prefix}_populate.sql, not {prefix}.sql core.py Accepted Fixed in af31e3443
2 Add fail-fast guard for an archetype missing #+data_source: generate.py Accepted Guard added in af31e3443 (clear error vs IsADirectoryError)
3 Currency description still says asset_class (schema column is monetary_nature) ores.sql.populate{.currency}.org Accepted Updated to monetary_nature in af31e3443; {{asset_class}} data lookup + CTE alias unchanged
4 Early return 1 leaves partial output on a mid-run failure generate.py Declined Pre-existing, non-blocking; left for a future cleanup

8. Result

Shipped in PR #1369. Populate/seed generation now runs through the ores.* physical-space graph, retiring the broken legacy model.json batch path:

  • load_org_dataset_model -> {dataset: {name, prefix}}, routed in load_model on #+type: ores.codegen.dataset; resolve_targets threads each archetype's #+data_source: into its unit; _generate_single renders dataset units from dataset_dir/data_source via the existing per-file generate_from_model (is_processing_batch=True suppresses the model.json batch dispatch), reusing all enrichment unchanged.
  • 9 ores.sql.populate archetypes carry #+data_source:; dataset_overview.org promoted to the dataset model (drawer opts the default-off facet in); seeder.sh repointed at codegen.sh generate.

Acceptance met: each dataset is an org model whose drawer enables its ores.sql.populate.* archetypes and codegen produces the populate SQL via the graph; archetypes are addressed, default-off nodes; resolve_output_path has a dataset branch ({dataset}/{prefix}). (get_template_mappings et al. removal is the legacy-path retirement tracked under the parent story's later steps.)

Beyond the original scope, two real defects were found and fixed:

  1. slovaris/solvaris dir divergence — standardised to a single populate/solvaris/; master renamed \{prefix}.sql -> \{prefix}_populate.sql (the convention every sibling master and catalogues_populate.sql already use); stale slovaris/ dir deleted.
  2. Schema drift (surfaced by the DB-load verification) — the currency template emitted column asset_class, but the schema had renamed it to monetary_nature. A full static cross-check confirmed it was the only mismatch across all 8 archetypes; fixed in the literate template.

Verified: compass db recreate -y -k loads the seed clean (100 currencies + 100 countries); 30 codegen tests pass (added tests/test_data_scope.py). Review round 1 (2 bot passes, design approved): 3 minor items accepted (stale comment, data_source fail-fast guard, asset_class->monetary_nature description), 1 pre-existing observation declined.

Emacs 29.3 (Org mode 9.6.15)