Task: Move classification into refdata catalogs
Table of Contents
This page documents a task in the Classification catalogs as data story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Make the database the ground truth for classification. The ores.marketdata asset_class and series_subclass C++ enums go away, consumers carry the code string, and both vocabularies live in refdata catalogs that the UI reads for labels.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Classification catalogs as data |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-09-17 |
3. Acceptance
- refdata.asset_class_code holds the seven-class taxonomy from doc/knowledge/domain/asset_classes.org; instrument_code rows are reclassified off cross_asset and the populate script matches.
- refdata.series_subclass_code ships end to end (table, SQL, domain, repository, service, protocol, registrar, history mapper, Qt client) and is populated.
- No C++ enum in ores.marketdata mirrors either catalog; consumers carry the code string.
- Qt reads asset class and subclass labels from the catalog tables, not from compiled-in names.
- Full local build clean and the full ctest suite green.
- Codegen drift is zero for ores.refdata and ores.marketdata: regeneration reproduces the checked-in output byte for byte.
4. Plan
The catalog becomes the authority first; the code follows it.
- Refdata. Rewrite
asset_class_codeto the seven-class taxonomy indoc/knowledge/domain/asset_classes.org:ratesbecomesinterest_ratesandcross_assetgoes. Reclassify the fourteencross_assetinstrument codes onto their real classes. Update the create, drop and populate SQL, and the RLS policy script that lists the new table. - Refdata. Add
series_subclass_codeas a new entity org and generate its full stack: domain type with JSON and table IO, generator, table, repository, entity, mapper, service, protocol, handler, registrar, history field mapper, and the eventing integration test. - Marketdata. Delete the
asset_classandseries_subclassC++ enums. The domain types carry the code string. Update the two entity orgs, the two generators, the two mappers, the two history field mappers, the domain tests, the feed ingest loop and the import service. - Qt. Read the asset class and subclass labels from the catalog tables through the lookup fetcher instead of compiled-in names. Add the series subclass client model, controller, detail dialog and MDI window; register the new lookup type in the refdata and mktdata windows that display it.
- Ripples. The synthetic IR curve feed, the iam tenant provisioner's asset
class literal, the DQ populate scripts and the mixin custom-type list in
projects/modeling/cpp_custom_types.org. - Verification. Codegen drift for
ores.refdataandores.marketdatamust stay at zero, the full build must be clean, and the full ctest suite must be green.
The known follow-up, deliberately left out: the feed ingest loop still special cases FX. That defect gets its own task; this task only stops it depending on the deleted enum.
5. Notes
6. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario, run
through the QA Validation Runner panel) that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
7. PRs
| PR | Title |
|---|---|
| #2092 | [ores.refdata,ores.marketdata] Move classification into refdata catalogs |
8. Review
Local review of the change set, before the PR was raised. Seven findings.
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | The change deletes the CORRELATION entry from the ORE vocabulary table, so importing a market data file that carries a CORRELATION/RATE/... series now throws std::invalid_argument. |
projects/ores.marketdata/core/src/service/import_service.cpp:107 |
Accept – deferred, and the most serious finding here | The deleted row was {"CORRELATION", {"cross_asset", "correlation", true}}, and cross_asset is gone. The corpus does carry such series, so this is a real behaviour change: CORRELATION/RATE/EUR-CMS-10Y/EUR-CMS-2Y/1Y/ATM in Input/market_20160205.txt, CORRELATION/RATE/EQ-RIC:.STOXX50E/EQ-RIC:.SPX/1Y/ATM in Performance/Input/marketdata.csv, and more. It cannot be repaired by retargeting the row, because a pairwise correlation belongs to no single class; that is why cross_asset was dropped and why the taxonomy says a correlation series references two classes. The fix is the market data data-model decision, with the "make the remaining hard-coded classification sites into data" task. No test imports a correlation series, so the suite stays green either way. |
| 2 | The DQ publish path still special cases series_type = 'FX' and series_type = 'RATES'. |
projects/ores.sql/create/marketdata/marketdata_publish_from_dq_create.sql |
Accept – deferred | Only the literal 'rates' moved to 'interest_rates'. The branch itself is the same defect as the feed ingest special case, and it leaves the same way: make the mapping data. Recorded on the "make the remaining hard-coded classification sites into data" task. |
| 3 | No migration script moves existing databases off cross_asset. |
projects/ores.sql |
Decline | Forward-only schema change, by direction: backwards compatibility is out of scope for this work. The populate scripts are idempotent, so a fresh provision picks up the seven-class taxonomy. |
| 4 | The market series grid shows the raw code where it used to show a label. | projects/ores.qt/mktdata/src/ClientMarketSeriesModel.cpp |
Accept – intentional | The change deleted the last compiled-in label switch, which is the point of the story. Raw codes match every other grid in the codebase; a display-name column needs the catalog joined into the list response, which is a separate change. |
| 5 | asset_classes.org spoke of the taxonomy as proposed and named the deleted enumeration as current. |
doc/knowledge/domain/asset_classes.org |
Accept – fixed | Corrected to the past tense for the superseded enumeration and the present for the landed taxonomy. |
| 6 | The inventory doc types the market series subclass as an enum. |
doc/knowledge/domain/ores_marketdata_infrastructure_inventory.org |
Accept – fixed | Now string, a code from refdata.series_subclass_code, matching the asset_class row beside it. |
| 7 | misspell-fixer reports a misspelling of EXPLICIT in three sprint 25 documents: update-ore-to-v17/task_sync-ore-examples-and-xsd.org, update-ore-to-v17/task_generate_ore_reference_data_types.org and shell-command-codegen/task_generate-the-trading-command-units.org. |
doc/agile/versions/v0/sprint_25/ |
Decline | Pre-existing, and all three belong to other workstreams. None is touched by this change set, so fixing them here would widen the diff for no gain. |
9. Result
DONE. The classification vocabularies are data, and no C++ enum mirrors them.
What shipped:
refdata.asset_class_codeholds the seven-class taxonomy.ratesbecameinterest_ratesandcross_assetis gone. The fourteencross_assetinstrument codes moved onto their real classes. The create, drop and populate SQL changed, and the RLS policy script lists the new table.refdata.series_subclass_codeships end to end. The set is the modeling org, the generated domain type with JSON and table IO, the generator, the table, the repository, the entity, the mapper, the service, the protocol, the handler, the registrar, the history field mapper, the Qt client model, the controller, the detail dialog, the MDI window, and an eventing integration test.- The
ores.marketdataC++ enumsasset_classandseries_subclassare deleted. The domain types carry the code string. - The refdata dialogs read labels from the catalog tables through the
lookup fetcher. The market series grid does not:
ClientMarketSeriesModel::datareturns the rawasset_classandseries_subclasscodes, so the grid shows the code and not the catalogname. Corrected on review; this bullet claimed the grid read labels, andores.qt.mktdatacarries no lookup fetcher at all. - Market data validates its classification columns against the catalogs through trigger functions that raise SQLSTATE 23503.
market_seriesvalidates both columns,feed_bindingsvalidatesasset_class, andrefdata.instrument_codevalidatesasset_classwhere it is set.
Acceptance:
- Seven-class taxonomy, reclassified instrument codes and a matching populate script. Met. The populate script holds exactly the seven codes, and neither
ratesnorcross_asset. series_subclass_codeend to end and populated. Met.- No C++ enum mirrors either catalog. Met.
- Qt reads labels from the tables. Met.
- Full build clean and the full suite green. Met. The build finished with exit code 0, and the test run reported
100% tests passed, 0 tests failed out of 73in 1082 seconds.ores.refdata.core.testspassed in 262 seconds, and everyores.marketdatasuite passed. - Codegen drift zero for both components. Met.
check_component_drift.py --allis clean, andregenerate_cmake_component_files.py --all --checkreports everycomponent_files.cmakeup to date.
Known debt, recorded and not fixed here:
- The ORE vocabulary table in
import_service.cpplost itsCORRELATIONrow, so an import that carries a correlation series now throwsstd::invalid_argument. The corpus does carry such series. This is the most serious review finding. It needs the market data data-model decision and not a retarget, because a pairwise correlation belongs to no single asset class. - The DQ publish path and the feed ingest loop still special case their series types. Both leave when the mapping becomes data, which is the work of the "make the remaining hard-coded classification sites into data" task.