Task: Declare the asset class lists once
Table of Contents
This page documents a task in the Clean ores.marketdata to the component clean standard story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
One hand-written list per asset-class axis, and a build that fails when any derived copy drifts from it. The refdata product taxonomy and the oresmd market-data namespace stop being two sets of hand-copied prose and numbers.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Clean ores.marketdata to the component clean standard |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-09-30 |
3. Acceptance
- One org model declares the refdata product taxonomy, the oresmd market-data namespace, and the mapping between them. No other file in the repository holds a hand-written copy of either list.
- The refdata seed SQL, the DQ badge definitions and mappings, the code-domain description, the model briefs and the documentation list are generated from that model.
- A codegen run fails when an oresmd model file names an authority the namespace list does not hold.
- A test fails when the seeded asset-class rows differ from the model.
- Every drift the census found is fixed, or its deferral is recorded here with a reason.
4. Decided
The two lists are two axes, not one list copied twice. Captured at
An asset class is two lists, not one,
because the two share enough spellings to be mistaken for each other. The refdata
product taxonomy answers what a trade or a market series is, and it is
FK-validated: ores_refdata_validate_asset_class_code_fn rejects any code absent
from ores_refdata_asset_class_codes_tbl, and market_series_asset_classes,
feed_bindings, series_classification_rules and instrument_codes all call
it. The oresmd namespace answers which key grammar a market data item is written
in, and it is what an oresmd_uri's authority carries.
They overlap on five spellings and diverge on five names. ir and
interest_rates are one concept under two spellings, as are security and
bond. correlation, shape_profile and rating exist only in the namespace,
because a correlation between two assets and a shape profile are market data
rather than products. The classification rules already say so: SHAPE_PROFILE
is classed commodity and RATING is classed credit.
Power joins the namespace only. A power fixing series is classed commodity
and its oresmd_uri authority carries power. The refdata taxonomy is
unchanged; the classification rules gain the power-to-commodity mapping. This
keeps the product taxonomy answering the product question, and leaves "show me
the power series" as a query on the URI authority.
One codegen model, everything else generated. The project already generates populate scripts from models and mustache templates, so the taxonomy gets the same treatment rather than a fourth hand-written copy.
5. The duplication, as measured
This census was taken at 2868477485 by a read-only walk of the repository, and
it found the two lists written in more than forty places. They fall into four
groups.
| group | what it holds | written how |
|---|---|---|
| the refdata list | the seed SQL, the model brief, the generated header brief, and the DQ badge definitions, mappings and code-domain description | by hand, in six files, three of them staging mirrors |
| the namespace list | modeling/oresmd/model.org and the ten *_quote_type.org files |
by hand once, then regenerated into the identifier variant, the requirement variant, the parser dispatch, the resolver, the enums and the tests |
| the mapping | nothing declares it | the nearest thing is the ORE-series-type-keyed classification rules, which never name an authority |
| third vocabularies | FpML's six-code codelist, the legacy ORE eight (rates, cross_asset), product_type, and the series-subclass axis |
separate subjects that only overlap |
The namespace list is already in good shape: it is authored once in
model.org and regenerated everywhere. The refdata list is not, and the
aliasing between the two is not written down at all.
6. Notes
The blanket fixing rule is the defect that makes this urgent. The single
FIXING row classifies every index fixing as interest_rates, and its
description reads "The index belongs to the rates class". That is true of the
interest-rate names and false of every other class: the corpus carries FX,
equity, inflation, commodity, power and security fixings. Until a fixing takes
its class from its index name, a power fixing series is mis-classed even after
the power authority exists.
The copies have already drifted. These are the defects the census found, and each is a place where a hand-kept copy stopped agreeing with its original.
ores.marketdata.pumlstill declaresenum asset_class { rates, ... }and thebond_market_data_identifierandrates_market_data_identifierstructs, all of which the code deleted.market_data_requirement.hpp's doc comment says "seven" while the variant beneath it lists ten classes.dq_badge_definitions_artefact_populate.sqlsays 56 rows in its header and holds 107;dq_badge_mappings_artefact_populate.sqlsays 108 and holds 161. The mappings have drifted by five rows in both directions since the one-off extraction. The asset-class subsets are still exact duplicates.fpml_asset_class_artefact_populate.sqlcarries a generated banner but was hand-extended with Inflation and Bond, so regenerating it from the vendored codelist would drop two rows.ore_asset_class_artefact_populate.sqluses the pre-rename spellingsratesandcross_asset.series_subclassis a three-way mismatch: the seed SQL holds 18 codes, the org model and the C++ sampler hold 15, and the classification rules use three the model omits (prepayment,transition_probability,index_fixing).series_classification_rule.organdfeed_binding.orghold byte-identical seven-element arrays for their test-data samplers.- The oresmd reject-URI list is hand-written in the template, in the tangle source beside it, and in the generated test.
The DQ artefact files are publish staging, not dead copies.
dq_badge_system_populate.sql seeds the system tenant's own tables and is
executed by dq_populate.sql. The *_artefact_populate.sql files populate the
staging tables the publish pipeline copies into another tenant, and their
headers say they were extracted programmatically from the system file when
first written and have been kept in sync by hand since. No generator exists in
the tree, which is why they drifted.
A correction to the premise that started this. market_series_asset_classes
does not declare a SQL foreign key to the taxonomy. There is no references
clause; the model calls it a soft FK and the insert trigger is what enforces it.
That matters because it means the enforcement point is one function, and a
generated catalogue satisfies it without a schema change.
7. Plan
- Declare the taxonomy. A model file holding each asset class once: code, name, description, display order, and the oresmd authority it answers to where it has one.
- Generate the seed SQL from it, replacing
refdata_asset_class_codes_populate.sql's hand-written values block. - Generate the DQ rows: the
asset_class_*badge definitions, the badge mappings, and the code-domain description, into both the system seed and the staging mirrors, so the hand sync ends. - Derive the briefs and the documentation list from the model, so the
prose copy in
asset_class_code.org, the generated header and the knowledge docs cannot drift. - Validate the namespace. The loader reads the model and fails when an oresmd model file names an authority it does not hold, naming the file and the authority.
- Make a fixing's class follow its index name. Add an index-name source to
series_classification_rulesalongsideliteralandcorrelation_operands, soFIXINGstops being a blanketinterest_rates. - Fix the drift, or record here why a case is deferred.
8. Measurements
Every number this task claims about a tree, with the command that produced it
and the commit it was measured at. A number without both is not a measurement,
because a clean-up record describes a tree and trunk moves under a branch: a
consumer count, a gate's component count and a suite's size all change with no
code changing, and the reader of a bare number cannot tell which tree it came
from. Where the raw output matters, put it in an evidence/ file beside the
task and name that file in the Command column.
| What | Value | Command | Commit |
|---|---|---|---|
| Places either list is written | 73 findings across five vocabularies | a read-only walk of the repository, reported in full in the session record | 2868477485 |
| refdata list copies | 6 files hold the values; 36 findings name refdata codes | the same walk | 2868477485 |
| namespace list copies | authored in model.org; 8 findings are its regenerated output |
the same walk | 2868477485 |
| Declared oresmd-to-refdata mapping | none | the same walk | 2868477485 |
| DQ badge rows, header claim vs actual | definitions 56 vs 107; mappings 108 vs 161 | wc -l and a row count of the two populate scripts |
2868477485 |
| DQ mapping rows drifted | 5, in both directions | a diff of the system seed against the staging mirror | 2868477485 |
9. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
10. PRs
| PR | Title |
|---|---|
| #2445 | [marketdata,refdata] Generate the taxonomy codes the test samplers read |
| #2444 | [refdata] Stop the entity model restating the taxonomy |
| #2443 | [refdata,codegen] Generate the asset-class DQ rows from the catalogue |
| #2440 | [marketdata,refdata] Close the series_subclass gap and settle the census's drifts |
| #2437 | [marketdata,refdata] Let a fixing take its class from its index name |
| #2432 | [refdata,codegen] Declare both asset-class lists in one model |
11. Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|
12. Result
A first unit declared the catalogue and generated the taxonomy seed from it. The
one model is ores.refdata.asset_class_catalogue. Its Taxonomy section holds the
seven product classes in display order, each with the prose the seed SQL carried; its
Namespace section holds the twelve oresmd authorities, each with the refdata class it
maps onto, so ir and interest_rates, security and bond, and the
namespace-only correlation and generic are written down once. The loader reads the
mapping into authority_to_class, and the oresmd batch loader now refuses a spec whose
authority the namespace does not hold – and an authority the namespace declares with no
spec – naming the file and the authority.
The seed is generated. A new ores.sql.asset_class facet and its codes_populate
archetype render refdata_asset_class_codes_populate.sql from the model's taxonomy
table; the generated file differs from the hand-written one by its AUTO-GENERATED
banner alone. Measured on the branch: the drift check reports no drift for refdata and
none across the catalogue, the codegen suite passes 1039, and the codegen, SQL and
documentation gates all pass.
What remains: the DQ asset-class rows, the briefs and documentation list, a fixing's class following its index name, and the census's drifts.
A second unit made a fixing take its class from its index name. The one FIXING
rule classified every index fixing as interest_rates, which is true of the rates names
and false of the equity, commodity, power, inflation and security fixings the corpus
carries – the defect the task calls the reason it is urgent. The rule now names a third
asset_class_source, index_name: the qualifier is the index name, the name projects
onto an oresmd identifier, and the catalogue's mapping turns that identifier's authority
into a refdata class. The mapping reaches C++ as a generated header in
ores.marketdata.api, written from the same model by a new ores.cpp.asset_class facet;
an authority the catalogue maps to no class, such as a correlation or the generic
wrapper, answers nothing, and a name no class can name leaves the fixing's class list
empty rather than guessing.
The table's check constraint and the entity's domain doc are regenerated, and the seed
row is index_name with a null class. Measured on the branch, on a database recreated
from the scripts: the classifier suite's corpus walk still classifies all 158 fixing
keys and its new cases read USD-SOFR as interest rates and an unnameable name as no
class; the workflow suite's recovery case flaked once under the parallel run and passed
on its own, which is a timing test unrelated to this change; the drift check reports no
drift for marketdata and none across the catalogue; and the codegen, SQL and
documentation gates all pass.
The census's drifts, each fixed or deferred with its reason. The census found seven, and re-reading the tree against them is what decides which still stand.
- The
series_subclassthree-way mismatch. Fixed. The seed carries eighteen codes and the classification rule model's sampler named fifteen;prepayment,transition_probabilityandindex_fixingwere missing, and the rules use the last of them forFIXING, so a generated sampler could emit a code the catalogue does not hold. Both the sampler and the code table's doc examples now carry all eighteen. - The
market_data_requirement.hpp"seven" against ten. No longer stands. The file's variant lists twelve alternatives and carries no count claim, so the census's finding was measured before a later change and has nothing left to fix. - The stale
asset_classinores.marketdata.puml. Deferred. The component diagram is hand-authored, and it is stale in several places at once: anenum asset_classthe code deleted, anasset_classclass beside it, and correlation identifier fields the struct no longer carries. A refresh pass has to redraw it against the current types and re-render its PNG, which is its own unit rather than a line edit here. - The DQ artefact headers' counts. Deferred to the unit that generates the DQ rows. The headers claim 56 definitions, 108 mappings and 31 code domains against 107, 161 and 40; rewriting the numbers by hand would only reset the same clock, and the generation that ends the hand sync is the next unit.
- The
ore_asset_classratesandcross_assetspellings. Deferred, because the census's reading is wrong. The file declares ORE's own eight-code asset-class vocabulary under theORE_ASSET_CLASScoding scheme, whereratesandcross_assetare ORE's spellings rather than stale copies of the refdata taxonomy. Whether that artefact should carry the refdata codes instead is a mapping decision the catalogue now makes possible but does not settle. - The byte-identical sampler arrays in two models. Deferred. The classification rule and feed binding models each carry the same seven-element array in their generator blocks. Sharing it means a codegen construct the tree does not have – a generated constant both read, or a field group – and the array is test data rather than the taxonomy, so the duplication is recorded rather than refactored under this task.
- The oresmd reject-URI list written three times. No longer a drift. The two of the three are the template and its tangle source, which are the literate-codegen pair by construction, and the third is the test generated from them.
A third unit generated the DQ rows from the catalogue. The taxonomy's seven badge
definitions, its asset_class code domain and its seven badge mappings were hand-written
inside dq_badge_system_populate.sql's single DO block and copied by hand into three
staging mirrors, whose headers had already drifted from the files they described. The
catalogue now carries each class's badge colour, text colour, severity, css class and
sort order, and names the code domain its mappings hang off with {codes} where the
taxonomy's own codes go; two new archetypes of the ores.sql.asset_class facet render
the system rows and the staging rows, so the badge metadata has one author and the
mirrors cannot drift again.
The system fragment is included beside the badge system seed, and the staging fragment
after the three mirrors. The staging deletes are scoped to the taxonomy's own rows,
because a mirror's delete takes a whole dataset and an unscoped one would take every
other domain's rows with it. Measured on a recreated database: the system tables still
hold 108 definitions, 40 code domains and 159 mappings, the staging tables 107, 40 and
161, and the taxonomy's own counts are 7, 1 and 7 in both. The full ctest suite passes 89
of 89 targets, the codegen suite passes 1046, the drift check reports no drift for
refdata and none across the catalogue, and every codegen, SQL and documentation gate
passes.
A fourth unit stopped the entity model restating the taxonomy, and closes the task.
ores.refdata.asset_class_code's brief named all seven codes and its column carried them
again as examples, and both reach the generated header and the create script's column
comment. The brief now states the taxonomy without the list, the column points at the
catalogue, and the model's prose says the catalogue is the one declaration, so the entity
and its generated layer hold no second copy. The analysis document that decided the
taxonomy keeps its resulting-list table: it is the conclusion of a dated piece of
analysis rather than a copy of the source, and rewriting it from the model would lose the
reasoning around it.
With that, every acceptance is met. The drift check reports no drift for refdata and
none across the catalogue, the codegen suite passes 1046, validate_docs.sh passes all
41 components, and SQL hygiene, the schema validation, the populate references and the
CMake lists all pass.
The task was reopened once, and the reopen is the honest part of this record. It was
closed after the fourth unit with acceptance 1 unmet: the classification rule and feed
binding models each still carried the same seven-element array of codes in a test-data
generator block. The census had listed that duplication as a drift, and the disposition
recorded it as deferred on the grounds that it is test data rather than the taxonomy –
but acceptance 1 says no file outside the catalogue holds a hand-written copy of the
list, and a literal seven-element array is one whatever it feeds. The array is now
generated: the codes are part of the header ores.cpp.asset_class already writes into
ores.marketdata.api, in display order, and both generator blocks read it through the
domain header they include. A class added to the catalogue now reaches the samplers, and
the drift is fixed rather than deferred.
Verified on the branch: the marketdata core suite's classification walk passes its 557
assertions in 19 cases, the codegen suite passes 1046, the full ctest suite passes 89 of
89 targets, the drift check reports no drift for marketdata and for refdata and none
across the catalogue after a full dry run, and every codegen, SQL and documentation gate
passes.