Task: Declare the asset class lists once

Table of Contents

This page documents a task in the Clean ores.marketdata to the component clean standard story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

One hand-written list per asset-class axis, and a build that fails when any derived copy drifts from it. The refdata product taxonomy and the oresmd market-data namespace stop being two sets of hand-copied prose and numbers.

2. Status

Field Value
State DONE
Parent story Clean ores.marketdata to the component clean standard
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-09-30

3. Acceptance

  • One org model declares the refdata product taxonomy, the oresmd market-data namespace, and the mapping between them. No other file in the repository holds a hand-written copy of either list.
  • The refdata seed SQL, the DQ badge definitions and mappings, the code-domain description, the model briefs and the documentation list are generated from that model.
  • A codegen run fails when an oresmd model file names an authority the namespace list does not hold.
  • A test fails when the seeded asset-class rows differ from the model.
  • Every drift the census found is fixed, or its deferral is recorded here with a reason.

4. Decided

The two lists are two axes, not one list copied twice. Captured at An asset class is two lists, not one, because the two share enough spellings to be mistaken for each other. The refdata product taxonomy answers what a trade or a market series is, and it is FK-validated: ores_refdata_validate_asset_class_code_fn rejects any code absent from ores_refdata_asset_class_codes_tbl, and market_series_asset_classes, feed_bindings, series_classification_rules and instrument_codes all call it. The oresmd namespace answers which key grammar a market data item is written in, and it is what an oresmd_uri's authority carries.

They overlap on five spellings and diverge on five names. ir and interest_rates are one concept under two spellings, as are security and bond. correlation, shape_profile and rating exist only in the namespace, because a correlation between two assets and a shape profile are market data rather than products. The classification rules already say so: SHAPE_PROFILE is classed commodity and RATING is classed credit.

Power joins the namespace only. A power fixing series is classed commodity and its oresmd_uri authority carries power. The refdata taxonomy is unchanged; the classification rules gain the power-to-commodity mapping. This keeps the product taxonomy answering the product question, and leaves "show me the power series" as a query on the URI authority.

One codegen model, everything else generated. The project already generates populate scripts from models and mustache templates, so the taxonomy gets the same treatment rather than a fourth hand-written copy.

5. The duplication, as measured

This census was taken at 2868477485 by a read-only walk of the repository, and it found the two lists written in more than forty places. They fall into four groups.

group what it holds written how
the refdata list the seed SQL, the model brief, the generated header brief, and the DQ badge definitions, mappings and code-domain description by hand, in six files, three of them staging mirrors
the namespace list modeling/oresmd/model.org and the ten *_quote_type.org files by hand once, then regenerated into the identifier variant, the requirement variant, the parser dispatch, the resolver, the enums and the tests
the mapping nothing declares it the nearest thing is the ORE-series-type-keyed classification rules, which never name an authority
third vocabularies FpML's six-code codelist, the legacy ORE eight (rates, cross_asset), product_type, and the series-subclass axis separate subjects that only overlap

The namespace list is already in good shape: it is authored once in model.org and regenerated everywhere. The refdata list is not, and the aliasing between the two is not written down at all.

6. Notes

The blanket fixing rule is the defect that makes this urgent. The single FIXING row classifies every index fixing as interest_rates, and its description reads "The index belongs to the rates class". That is true of the interest-rate names and false of every other class: the corpus carries FX, equity, inflation, commodity, power and security fixings. Until a fixing takes its class from its index name, a power fixing series is mis-classed even after the power authority exists.

The copies have already drifted. These are the defects the census found, and each is a place where a hand-kept copy stopped agreeing with its original.

  • ores.marketdata.puml still declares enum asset_class { rates, ... } and the bond_market_data_identifier and rates_market_data_identifier structs, all of which the code deleted.
  • market_data_requirement.hpp's doc comment says "seven" while the variant beneath it lists ten classes.
  • dq_badge_definitions_artefact_populate.sql says 56 rows in its header and holds 107; dq_badge_mappings_artefact_populate.sql says 108 and holds 161. The mappings have drifted by five rows in both directions since the one-off extraction. The asset-class subsets are still exact duplicates.
  • fpml_asset_class_artefact_populate.sql carries a generated banner but was hand-extended with Inflation and Bond, so regenerating it from the vendored codelist would drop two rows.
  • ore_asset_class_artefact_populate.sql uses the pre-rename spellings rates and cross_asset.
  • series_subclass is a three-way mismatch: the seed SQL holds 18 codes, the org model and the C++ sampler hold 15, and the classification rules use three the model omits (prepayment, transition_probability, index_fixing).
  • series_classification_rule.org and feed_binding.org hold byte-identical seven-element arrays for their test-data samplers.
  • The oresmd reject-URI list is hand-written in the template, in the tangle source beside it, and in the generated test.

The DQ artefact files are publish staging, not dead copies. dq_badge_system_populate.sql seeds the system tenant's own tables and is executed by dq_populate.sql. The *_artefact_populate.sql files populate the staging tables the publish pipeline copies into another tenant, and their headers say they were extracted programmatically from the system file when first written and have been kept in sync by hand since. No generator exists in the tree, which is why they drifted.

A correction to the premise that started this. market_series_asset_classes does not declare a SQL foreign key to the taxonomy. There is no references clause; the model calls it a soft FK and the insert trigger is what enforces it. That matters because it means the enforcement point is one function, and a generated catalogue satisfies it without a schema change.

7. Plan

  1. Declare the taxonomy. A model file holding each asset class once: code, name, description, display order, and the oresmd authority it answers to where it has one.
  2. Generate the seed SQL from it, replacing refdata_asset_class_codes_populate.sql's hand-written values block.
  3. Generate the DQ rows: the asset_class_* badge definitions, the badge mappings, and the code-domain description, into both the system seed and the staging mirrors, so the hand sync ends.
  4. Derive the briefs and the documentation list from the model, so the prose copy in asset_class_code.org, the generated header and the knowledge docs cannot drift.
  5. Validate the namespace. The loader reads the model and fails when an oresmd model file names an authority it does not hold, naming the file and the authority.
  6. Make a fixing's class follow its index name. Add an index-name source to series_classification_rules alongside literal and correlation_operands, so FIXING stops being a blanket interest_rates.
  7. Fix the drift, or record here why a case is deferred.

8. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
Places either list is written 73 findings across five vocabularies a read-only walk of the repository, reported in full in the session record 2868477485
refdata list copies 6 files hold the values; 36 findings name refdata codes the same walk 2868477485
namespace list copies authored in model.org; 8 findings are its regenerated output the same walk 2868477485
Declared oresmd-to-refdata mapping none the same walk 2868477485
DQ badge rows, header claim vs actual definitions 56 vs 107; mappings 108 vs 161 wc -l and a row count of the two populate scripts 2868477485
DQ mapping rows drifted 5, in both directions a diff of the system seed against the staging mirror 2868477485

9. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

10. PRs

PR Title
#2445 [marketdata,refdata] Generate the taxonomy codes the test samplers read
#2444 [refdata] Stop the entity model restating the taxonomy
#2443 [refdata,codegen] Generate the asset-class DQ rows from the catalogue
#2440 [marketdata,refdata] Close the series_subclass gap and settle the census's drifts
#2437 [marketdata,refdata] Let a fixing take its class from its index name
#2432 [refdata,codegen] Declare both asset-class lists in one model

11. Review

Comment summary File Decision Notes
       

12. Result

A first unit declared the catalogue and generated the taxonomy seed from it. The one model is ores.refdata.asset_class_catalogue. Its Taxonomy section holds the seven product classes in display order, each with the prose the seed SQL carried; its Namespace section holds the twelve oresmd authorities, each with the refdata class it maps onto, so ir and interest_rates, security and bond, and the namespace-only correlation and generic are written down once. The loader reads the mapping into authority_to_class, and the oresmd batch loader now refuses a spec whose authority the namespace does not hold – and an authority the namespace declares with no spec – naming the file and the authority.

The seed is generated. A new ores.sql.asset_class facet and its codes_populate archetype render refdata_asset_class_codes_populate.sql from the model's taxonomy table; the generated file differs from the hand-written one by its AUTO-GENERATED banner alone. Measured on the branch: the drift check reports no drift for refdata and none across the catalogue, the codegen suite passes 1039, and the codegen, SQL and documentation gates all pass.

What remains: the DQ asset-class rows, the briefs and documentation list, a fixing's class following its index name, and the census's drifts.

A second unit made a fixing take its class from its index name. The one FIXING rule classified every index fixing as interest_rates, which is true of the rates names and false of the equity, commodity, power, inflation and security fixings the corpus carries – the defect the task calls the reason it is urgent. The rule now names a third asset_class_source, index_name: the qualifier is the index name, the name projects onto an oresmd identifier, and the catalogue's mapping turns that identifier's authority into a refdata class. The mapping reaches C++ as a generated header in ores.marketdata.api, written from the same model by a new ores.cpp.asset_class facet; an authority the catalogue maps to no class, such as a correlation or the generic wrapper, answers nothing, and a name no class can name leaves the fixing's class list empty rather than guessing.

The table's check constraint and the entity's domain doc are regenerated, and the seed row is index_name with a null class. Measured on the branch, on a database recreated from the scripts: the classifier suite's corpus walk still classifies all 158 fixing keys and its new cases read USD-SOFR as interest rates and an unnameable name as no class; the workflow suite's recovery case flaked once under the parallel run and passed on its own, which is a timing test unrelated to this change; the drift check reports no drift for marketdata and none across the catalogue; and the codegen, SQL and documentation gates all pass.

The census's drifts, each fixed or deferred with its reason. The census found seven, and re-reading the tree against them is what decides which still stand.

  • The series_subclass three-way mismatch. Fixed. The seed carries eighteen codes and the classification rule model's sampler named fifteen; prepayment, transition_probability and index_fixing were missing, and the rules use the last of them for FIXING, so a generated sampler could emit a code the catalogue does not hold. Both the sampler and the code table's doc examples now carry all eighteen.
  • The market_data_requirement.hpp "seven" against ten. No longer stands. The file's variant lists twelve alternatives and carries no count claim, so the census's finding was measured before a later change and has nothing left to fix.
  • The stale asset_class in ores.marketdata.puml. Deferred. The component diagram is hand-authored, and it is stale in several places at once: an enum asset_class the code deleted, an asset_class class beside it, and correlation identifier fields the struct no longer carries. A refresh pass has to redraw it against the current types and re-render its PNG, which is its own unit rather than a line edit here.
  • The DQ artefact headers' counts. Deferred to the unit that generates the DQ rows. The headers claim 56 definitions, 108 mappings and 31 code domains against 107, 161 and 40; rewriting the numbers by hand would only reset the same clock, and the generation that ends the hand sync is the next unit.
  • The ore_asset_class rates and cross_asset spellings. Deferred, because the census's reading is wrong. The file declares ORE's own eight-code asset-class vocabulary under the ORE_ASSET_CLASS coding scheme, where rates and cross_asset are ORE's spellings rather than stale copies of the refdata taxonomy. Whether that artefact should carry the refdata codes instead is a mapping decision the catalogue now makes possible but does not settle.
  • The byte-identical sampler arrays in two models. Deferred. The classification rule and feed binding models each carry the same seven-element array in their generator blocks. Sharing it means a codegen construct the tree does not have – a generated constant both read, or a field group – and the array is test data rather than the taxonomy, so the duplication is recorded rather than refactored under this task.
  • The oresmd reject-URI list written three times. No longer a drift. The two of the three are the template and its tangle source, which are the literate-codegen pair by construction, and the third is the test generated from them.

A third unit generated the DQ rows from the catalogue. The taxonomy's seven badge definitions, its asset_class code domain and its seven badge mappings were hand-written inside dq_badge_system_populate.sql's single DO block and copied by hand into three staging mirrors, whose headers had already drifted from the files they described. The catalogue now carries each class's badge colour, text colour, severity, css class and sort order, and names the code domain its mappings hang off with {codes} where the taxonomy's own codes go; two new archetypes of the ores.sql.asset_class facet render the system rows and the staging rows, so the badge metadata has one author and the mirrors cannot drift again.

The system fragment is included beside the badge system seed, and the staging fragment after the three mirrors. The staging deletes are scoped to the taxonomy's own rows, because a mirror's delete takes a whole dataset and an unscoped one would take every other domain's rows with it. Measured on a recreated database: the system tables still hold 108 definitions, 40 code domains and 159 mappings, the staging tables 107, 40 and 161, and the taxonomy's own counts are 7, 1 and 7 in both. The full ctest suite passes 89 of 89 targets, the codegen suite passes 1046, the drift check reports no drift for refdata and none across the catalogue, and every codegen, SQL and documentation gate passes.

A fourth unit stopped the entity model restating the taxonomy, and closes the task. ores.refdata.asset_class_code's brief named all seven codes and its column carried them again as examples, and both reach the generated header and the create script's column comment. The brief now states the taxonomy without the list, the column points at the catalogue, and the model's prose says the catalogue is the one declaration, so the entity and its generated layer hold no second copy. The analysis document that decided the taxonomy keeps its resulting-list table: it is the conclusion of a dated piece of analysis rather than a copy of the source, and rewriting it from the model would lose the reasoning around it.

With that, every acceptance is met. The drift check reports no drift for refdata and none across the catalogue, the codegen suite passes 1046, validate_docs.sh passes all 41 components, and SQL hygiene, the schema validation, the populate references and the CMake lists all pass.

The task was reopened once, and the reopen is the honest part of this record. It was closed after the fourth unit with acceptance 1 unmet: the classification rule and feed binding models each still carried the same seven-element array of codes in a test-data generator block. The census had listed that duplication as a drift, and the disposition recorded it as deferred on the grounds that it is test data rather than the taxonomy – but acceptance 1 says no file outside the catalogue holds a hand-written copy of the list, and a literal seven-element array is one whatever it feeds. The array is now generated: the codes are part of the header ores.cpp.asset_class already writes into ores.marketdata.api, in display order, and both generator blocks read it through the domain header they include. A class added to the catalogue now reaches the samplers, and the drift is fixed rather than deferred.

Verified on the branch: the marketdata core suite's classification walk passes its 557 assertions in 19 cases, the codegen suite passes 1046, the full ctest suite passes 89 of 89 targets, the drift check reports no drift for marketdata and for refdata and none across the catalogue after a full dry run, and every codegen, SQL and documentation gate passes.

Emacs 29.3 (Org mode 9.6.15)