Story: Complete the market series classification
Table of Contents
This page documents a story in Sprint 25. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
1. Goal
Every market-data series the ORE corpus carries imports without an unknown-type error. A test proves that, and it needs no database, no NATS and no import pipeline to run. A pairwise correlation records both of the asset classes it relates.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent sprint | Sprint 25 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-09-23 |
3. Acceptance
- Every distinct series type in
external/ore/examplesclassifies. - The classifier runs in a unit test with no service, database or network.
- A market series carries a set of asset classes, so a correlation can name the two it relates.
- A series has a point only if its key has a point dimension. No stored column claims otherwise.
4. Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Extract the series classifier and cover the whole ORE corpus | DONE | 2026-09-17 | 2026-09-18 | Lift the ORE-vocabulary mapping out of import_service.cpp into a standalone series_classifier class that tests can reach without a database, NATS or the import pipeline. Complete the vocabulary so all 42 series types the ORE examples carry classify. Add a corpus-wide test that walks external/ore/examples and asserts every distinct series key classifies. |
| Give a market series a set of asset classes | DONE | 2026-09-17 | 2026-09-18 | Replace the single not-null market_series.asset_class column with an ores_marketdata_market_series_asset_classes_tbl junction, so a pairwise correlation can reference the two classes it relates. Update the classifier's consumers, the DQ publish path and the Qt market series grid, and confirm the refdata catalogue descriptions read correctly. |
| Read and write the market series asset classes in Qt | ABANDONED | 2026-09-23 | The Qt client reads and writes the market_series_asset_classes junction, so the market series grid shows the asset classes each series belongs to and the New Series form records them. Junction protocols are hand-written, so this carries its own protocol, client fetch and in-memory join. | |
| Register the unmapped series types in the key registry | DONE | 2026-09-18 | 2026-09-18 | Add series_key_registry entries for the eight ORE series types the corpus carries but the registry does not know: OI_FUTURE, SHAPE_PROFILE, BOND_OPTION, RATING, INDEX_CDS_TRANCHE, GENERIC-MD, CPR and FIXING. Without an entry a key folds its whole remainder into the qualifier and carries no point, so every distinct key becomes its own series with one observation. |
| Complete the market series asset class junction | DONE | 2026-09-20 | 2026-09-20 | The market_series_asset_classes junction carries neither the row level security its sibling data tables have nor a drop script. The junction template ignores the rls_tenant_isolation flag, so no junction in the tree is tenant isolated, and the table is absent from marketdata_drop.sql even though refdata hand-writes a drop file for its own junction. |
| Handle compound keys in the generated eventing artefacts | ABANDONED | 2026-09-23 | Two generated eventing artefacts mis-handle a compound natural key. The changed event template names the key vector after the first key column and pluralises it, so the series classification rule event carries series_types holding both the type and the metric. The generated eventing test matches a notification as a set rather than as an exact key, so a permutation of the key parts would also match. |
5. Decisions
5.1. The ORE vocabulary is a managed table, not a compiled list
The series key shape and the classification rule both become rows. A compiled list cannot serve the case that matters: when ORE adds a type, or a user brings one of their own, the answer must be an inserted row rather than a rebuild. The tables are seeded from the corpus, so the rows an environment receives are the rows the tests classify against.
5.2. Two tables, two owners
The shape table belongs to ores.ore, because it records ORE's own
key grammar, which ORE defines and we only read. The classification
table belongs to ores.marketdata, because it maps a type onto refdata
taxonomy codes, which are ours. The two facts change for different
reasons, so they live apart. Each component that owns an entity model
owns a schema, so ores.ore holding tables needs no exception.
5.3. The reader takes the table
series_key_registry and series_classifier are values built from
rows. Each is built once per import and passed down by const reference,
so a reader never reaches for the database and a test runs with no
database, no NATS and no pipeline. There is no process-wide cache to
invalidate when a row changes.
5.4. An empty table throws; an unknown type folds
An empty table is a configuration fault and throws, naming the table, because it would silently degrade every key into a series of its own. A type with no row is a different case: its key folds whole into the qualifier, loses nothing, and still reconstructs verbatim, so an uncatalogued type never takes an import down. The alarm lives in the corpus test, which is red whenever the corpus carries a type with no row.
5.5. Row level security is opted into per table
Every tenant-scoped table enables row level security in its own model,
through the :rls_tenant_isolation: flag, and the generated policy
reads ores_iam_current_tenant_id_fn(). Both tables this story adds
carry it, and the junction carries it too: the junction create
archetype now reads the flag, so a junction opts in exactly the way a
domain entity does. The market series asset class junction is the only
junction that needed to opt in, because the iam account party junction
already has its policy in iam_rls_policies_create.sql and opting in
again would have emitted a second one. No junction drop archetype
exists, so a junction's drop file stays hand-written and is wired into
its component's _drop.sql by hand.
5.6. The correlation operand rule stays compiled
An operand's asset class comes from the prefix ORE gives it, EQ-,
FX- and COMM-, and from the -CMS- a rates operand carries instead
of a prefix. That is a naming convention about ORE's own keys, which no
row of ours can state, so it stays in code. Task 235 documents it.
6. Out of scope
7. Result
Closed at sprint 25 close. The classifier, the corpus coverage and the asset class junction shipped. The compound-key eventing task goes to the sprint 26 codegen pass, and the Qt task lost its client.