Task: Bring ores.marketdata to the clean standard
Table of Contents
This page documents a task in the Clean ores.marketdata to the component clean standard story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
ores.marketdata passes the Component Clean Standard for a component of kind Entity.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Clean ores.marketdata to the component clean standard |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-01 |
3. Acceptance
- Every checklist item that applies to a component of kind Entity passes, with its evidence recorded on the task.
- Every item that does not apply is recorded as not applicable, with the reason.
- Every generator defect found is fixed at its source, with a regression test.
- ores.marketdata is listed in
COMPONENTS_UNDER_TEST, andcheck_component_drift.py --allpasses.
4. Plan
Work the Component Clean Standard in its section order, one entity at a time, and record each item's result in the table below.
- Baseline and inventory (B01 to B05). Record the baseline before any edit.
- Models (M01 to M10).
- Protocol (P01 to P06).
- Generation (G01 to G06).
- Wiring and data (W01 to W04).
- Shell (S01, S02).
- Structure and hygiene (H01 to H03).
- Verification (V01 to V06), ending with the registry entry.
Corrected on 2026-10-04 by Correct the marketdata clean-standard record, which re-measured all 49
items at ec37b05f77 and holds the current record. The rows below describe the
tree when this task closed. Where the re-measurement found a row wrong, the row
says so.
| Item | Result | Evidence |
|---|---|---|
| B01 | Pass | Baseline at origin/main 8a16c7a4f0: check_component_drift.py --component marketdata --dry-run exits 1 with 82 files to change and 14 to create. Full output kept on the task's evidence. |
| B02 | Pass | 19 model files in four metatypes: 6 ores.codegen.entity, 1 ores.codegen.junction, 8 ores.codegen.oresmd_quote_type (a metatype of its own, no variability), 4 ores.codegen.component. Survey under "Model survey" below. |
| B03 | Pass | 285 C++ files: 188 generated, 97 hand-written (20 tests, 77 not). The 77 divide into 24 generatable, 36 infrastructure that stays, 17 dead. Survey table on the task's evidence. |
| B04 | Pass | 12 protocol headers: 6 generated, 6 hand-written. Subject inventory on the task's evidence. |
| B05 | Pass | Captures read: the unwired series_classification_rule event mapping (marketdata never composes the series_classification_rule event registrar), the missing market_observation_repository tests (market_observation_repository has no test coverage), the RLS party-isolation stopgap (commit 83292021a added party_id to ores.marketdata), the timeseries metatype question (timeseries entity codegen analysis). No in-flight branch collides: feature/identity-core-series-and-bindings is abandoned and holds PR 5's work. |
| B06 | Pass | 49 public symbols with a production consumer, 19 test-only, 4 with none. Test-only counts as dead. The oresmd layer's test-only symbols were the one recorded exception, and PR 4 closed it rather than keeping it: import_service.cpp now runs every parsed datum through from_ore_key() -> market_data_identifier -> to_quote_key() and builds the series and observation rows from the registry's decomposition of that canonical spelling. A key oresmd cannot name – BOND, the option and capfloor families – keeps the parser's own decomposition and is imported anyway, because ORE's examples carry such keys. The FX/RATE reversed-pair correction moved with it: it used to find the slash in the qualifier and swap two currencies by hand, and now from_ore_key(key, checker) swaps the identifier's pair field. Proved on the corpus, not on the two hand-written lines the import suite feeds it: all 118 market payload files still import and read back under the hidden .corpus-full tag. |
| M01 | Pass | No legacy JSON, no lookup_entity, no retired table type; all 15 entity and junction models load on the current format. |
| M02 | Pass | Four models bound and feed_binding recorded as a no-match with its actual feature values and the reason. check_model_drift.py exits 0. |
| M03 | Pass | The junction's restated :has_tenant_id: true removed, together with the same restatement on the four newly bound models. |
| M04 | Pass | Every stated feature sits in the namespace the loader reads. |
| M05 | Pass | All seven models describe one real table; none is joined or denormalised; all seven tables carry history, so the current-state shape does not apply. |
| M06 | Pass (corrected) | Six hand-written protocol headers carry wire types no model covers: crm_protocol.hpp, curve_republish_protocol.hpp, curve_snapshot_protocol.hpp, import_protocol.hpp, market_feed_config_protocol.hpp, market_series_export_protocol.hpp. Sized in PR 3 and moved to a follow-up: 26 structs and 85 fields over six ores.codegen.operation models, plus one generator feature they cannot land without. std::map has no TypeScript projection in org_loader._ts_type, and market_feed_config_protocol.hpp carries std::map<std::string, feed_kind_counts> by_kind and std::map<std::string, int> stopped_by_kind, so the models cannot load until the map branch exists. The usual escape, :ores.ts.protocol.enabled: nil, is forbidden globally by the twin-coverage gate. It is also a cross-component header swap: the same six headers are consumed by ores.synthetic, ores.iam, ores.shell, ores.reporting and ores.ore. Not a regeneration item. Corrected 2026-10-04: all seven protocol headers are generated, operations_protocol.hpp from ores.marketdata.operations, so the item passed when the task closed. |
| M07 | Pass | Singular named, key column types real, tenant-scoped row-level security in the generated SQL. read_only, client_read_only and party_id_is_target are genuinely not applicable. |
| M08 | Pass | The one custom type, std::chrono::year_month_day, is bound to <chrono>; SQL spellings are canonical; enabled is a boolean. |
| M09 | Pass | The ten legacy Presentation properties naming retired message types are gone. Zero Qt or Wt references in any model; the 14 in hand-written C++ comments are H02 and belong to PR 3. |
| M10 | Pass | The component now has a root component_overview.org with #+component_kind: composite and #+parts: api client core service, so its root CMakeLists.txt is generated from cmake_composite_root.mustache and the subdirectory list is no longer hand-maintained. validate_docs.py reports all 40 components passing. |
| P01 | Fail | The models declare the full canonical verb set for all six entities and the committed tree now matches it – list_/get_/get_many_/put_/put_many_/delete_/delete_many_ plus the version reads. What is still absent is the point-in-time read, and PR 3 established that it is not the plural repositories. P01 is the canonical as_of field on the list request, driven by the model flag :has_as_of_lookup: true, which four refdata models set and no marketdata model does. The plural repositories' read_as_of is a different thing: financial valid-time curve reconstruction, filed under H02. Re-scoped, and deferred with H02. |
| P03 | Pass | The seven stale generated artefacts are regenerated, and the two unserved surfaces are served: register_observation_lineage_handlers and register_series_classification_rule_handlers are now composed by the umbrella registrar alongside the other four, so every entity family the models declare has a subscribed handler. |
| P04 | Pass (corrected) | No operation model anywhere. PR 3 sized it and folded it into M06: the same six headers, the same generator gap, the same follow-up. Corrected 2026-10-04: the operation model exists; same evidence as M06. |
| W01 | Pass | All six event mappings are composed at the service's composition point, all six handler registrars are composed by the umbrella registrar, and the four history-provider registrars are wired through register_history_handlers on a function-local dispatch_registry, following ores.refdata. The two no_audit_columns entities have no provider, and that is now stated where the wiring is rather than left unexplained: the facet is withheld for them because a version type has to carry an actor as well as the timestamp. |
| W02 | Fail | The check passes vacuously because marketdata is not in its registry. Read by hand: 16 codes are checked, none is seeded exactly as checked, and seven seeded marketdata:: codes are checked by no handler. |
| G03 | Pass | Clean in both directions: all 188 marked files name a template that exists, and no hand-written file carries the marker or a near-miss. |
| G05 | Pass | One generator defect found and fixed at its source with a regression test: the junction shell-command include. |
| G06 | Pass | All four modules' component_files.cmake are regenerated from the tree, and regenerate_cmake_component_files.py --component <part> --check reports every marketdata part up to date. The client's two lists also drop the 13 deleted CRM sources and tests. |
| H01 | Partial | The client diagram now carries the two-pass sentinel, so all four module diagrams follow the convention. The composite root still has none, and that is a deliberate stop rather than an oversight: modeling/ores.marketdata.puml is a hand-authored data-model diagram for the oresmd identifier grammar, not a scan of C++ sources, so the automated pass has nothing to contribute to a component whose root owns no code. Recorded here rather than forced. |
| H02 | Pass | The 13 dead client CRM files and their three test suites are deleted, and the six orphaned *_changed_event.hpp headers with them – generation stopped emitting them when the events moved beside their protocol, and the only remaining references were their own trait definitions. The two plural repositories are all that is left of the 17, and they are not dead: see P01. The seven Qt references in hand-written comments are rewritten. |
| H05 | Pass | client/modeling/component_overview.org is real prose describing the three things the part now holds, with a diagram drawn to the same two-pass convention as its siblings and a PNG to match. The MISSING_PUML ores.marketdata.client exception is removed from projects/ores.codegen/docs_exceptions.txt, and validate_docs.py passes for all 40 components. |
| V01 | Pass | All four test targets build (compass build cannot run here; cmake --build --preset linux-clang-debug-make used directly, see Notes). |
| V02 | Pass | api.tests 71 assertions in 14 cases, client.tests 6 in 6, service.tests 164 in 24, core.tests 1,161 in 296 – all green, the previously failing series_classification_rule_eventing_integration_tests included. The cause was the stale trigger, not the test, and re-applying the canonical function is what fixed it; the test itself is unchanged apart from the regenerate. Measured on a database rebuilt from this tree (compass db recreate -y -k), because the restored schema was a day behind and produced two failures that did not exist in the code – see the notes on schema drift. |
| V05 | Pass | Re-run when the component joined, at V06. check_component_drift.py --all --dry-run reports no drift; check_model_drift.py, check_handler_permissions.py (7 components, 290 codes), check_populate_references.py, check_route_auth_declarations.py, check_shell_menu_collisions.py and the shell and HTTP recipe inventories all exit 0; regenerate_cmake_component_files.py --all --check leaves marketdata current; validate_docs.sh passes all 41 components. |
| V06 | Not met (corrected) | The seven hand-written protocol headers are one operation model, ores.marketdata.operations, so all 13 of the component's protocol headers have a TypeScript twin. check_protocol_twin_coverage.py reports 201 headers in 13 committed components with no model opting out, and marketdata is listed in COMPONENTS_UNDER_TEST. Verified on the rebased branch, with the database recreated and the fleet started: the codegen suite passes 1032, all 35 local checks pass, the tree builds whole, and ctest passes 89 of 89 targets. Two trunk breaks the first run found were fixed rather than recorded: a stale log-filename expectation in ores.service.tests and the workflow component's CMake header list. Corrected 2026-10-04: the joining record had no accepted exception for the items it failed; marketdata left COMPONENTS_UNDER_TEST on 2026-10-04. |
| V07 | Pass | The vacuous tests were in the client CRM suite, which is deleted with H02. Nothing is left to vacuous-check. |
| V08 | Not done (corrected) | The end-to-end question is measured now. service_corpus_import_tests.cpp imports every market payload under external/ore/examples through the import service and reads the rows back; all 118 files pass, and each file's values match the ones the file carried. A bounded sample of the smallest payloads runs by default and the corpus-wide walk sits behind the .corpus-full hidden tag. The comment in that test went through a first draft that got two things wrong – the test tenant is per binary rather than per case, so the walk changed other cases' row counts, and a corpus key splits into the stored columns by per-asset-class rules, so looking a series up by rebuilding its key tests the split as much as the import. It builds a tenant of its own and compares a multiset of values instead. The walk found three defects on its first runs, all of them items the analysis had recorded as fixed in PR 2: parse_date read two of the five date formats the corpus uses, so DD-MM-YYYY, DD.MM.YY and DD.MM.YYYY threw and two 4,652-line files could not be imported at all; the tokeniser took its comma path only when a line held two commas or more, so a line like 2025-02-10,RECOVERY_RATE/RATE/BANK/SR/EUR 0.4 aborted its whole file; and the de-duplication the import performs by design had to be encoded in the expectation rather than read as data loss. Corrected 2026-10-04: this measures the corpus walk, not the per-file survey of src/ that V08 asks for. The corpus selection holds 99 market payload files at ec37b05f77, not 118. |
| Oresmd coverage | Overstated (corrected) | Re-measured on main after PR 3. The gate reports 43 ORE series types, 41 represented in oresmd and 2 recorded with a reason (FIXING and GENERIC-MD), zero shape mismatches, 42 oresmd quote types covering 38 distinct ORE types, and 6 of 6 asset classes declaring volatility wired. Two allowlists in oresmd_ore_coverage_tests.cpp are empty and asserted equal to empty: types_oresmd_cannot_name() and types_with_round_trip_gaps(). So no ORE type is left unnamed and no key projects without reading back. On line counts the corpus has grown since PR 2: 511,002 lines, 486,436 of them (95.2%) on modelled types. The remainder are lines whose first token is not a series type at all (Lufthansa, SP5, EURUSD, SwaptionVolatility), not market-data series. The earlier 485,606-of-485,606 figure was of the keys the gate expected to name; the two bases differ, and this row states both rather than picking the flattering one. CPR moved from commodity to the security class, where its ISIN-keyed shape belongs, and F-TIIE joined the index family as an overnight benchmark, which meant widening both CHECK constraints on ores_synthetic_ir_curve_generation_configs_tbl with a one-shot migration. Corrected 2026-10-04: this held only in memory; through the stored URI 7,254 keys did not read back. The hand-written codecs replaced the gate: every one of the 108,057 distinct corpus keys reads, writes back and round-trips through its URI. |
| Oresmd test coverage | Pass | All 42 modelled quote types carry both a projection test and a round-trip test. Six were missing the round trip and one the projection when the requirement was introduced; the projection gap was commodity.cpr, recorded as an exemption because the only key it could assert then was the wrong one. PR 2 moved CPR to the security class, where it asserts the ISIN-keyed shape it actually writes, so the exemption list is empty and the gate fails a stale exemption. The round-trip cases now also assert that the documented URI names a real ORE artefact: they checked URI-to-identifier stability alone, which passed for a URI that projected to nothing, so a stale example row could sit in a model undetected. Reverting one such row makes the case fail, which is how the guard was checked. |
5. Notes
The discovery behind this table is written up in full at
Analysis: where ores.marketdata stands, with the raw
surveys under tmp/marketdata_analysis/.
5.1. What the environment would not let us do
compass build cannot run in this environment: its systemd-run --user
--scope wrapper fails with "Failed to connect to user scope bus". The
same targets build through cmake --build --preset linux-clang-debug-make
directly, and that is how V01 and V02 were measured. The two suites that
read the tables need ORES_TEST_DB_USER from .env and a live database.
The TypeScript side cannot be checked here either. projects/ores.web is
installed with npm, and its node_modules has neither zod nor
@msgpack/msgpack, so tsc --noEmit reports module-resolution errors for
every file rather than type errors. The eight regenerated marketdata .ts
files are therefore recorded as rendered-but-unchecked. The codegen suite
does cover the TypeScript rendering, at 826 passing cases after the rebase.
5.2. One generator defect the rebase exposed
Rebasing onto main after the PR was raised picked up 48f4282df6, which made
the generated row-level-security policies idempotent so that re-running
setup_schema.sql against a populated database stops aborting on a policy
that already exists. It fixed the domain-entity archetype and missed the
junction one.
The guard was not missing from the tree, only from the generator: 02c78a7dda
had hand-added drop policy if exists to the checked-in junction SQL. So the
two disagreed, and regenerating this component – whose only junction opts
into tenant isolation – removed the guard and reintroduced the abort. The
fix is in the junction template, with a regression test that renders a
junction with :rls_tenant_isolation: and asserts the drop precedes the
create; removing the guard from the template makes it fail. No generated file
changes, which is the point: the checked-in SQL was already right and the
generator now agrees with it.
That same commit also fixed the reporting omission this PR's shell commit
originally carried. It found it independently and first, so the finding is
main's, not this PR's; only marketdata is added to the shell's parts here.
5.3. Check the database before believing a red suite
compass db status opens with the schema version and a drift line, and PR 3
is the argument for reading it first. The restored database was a day behind
this tree. Two suites went red on it, and neither failure was in the code:
ores.synthetic.core.testsfailed all eight of its eventing integration tests, because the restored schema still held the legacy notify function for those tables while the committed SQL was already canonical. A singleselect prosrc like '%entity_ids%' from pg_procsettled it in one query.ores.iam.core.testsfailed one case that writes an asset image, because the restoredores_assets_images_tblstill hadidwhere the tree now expectsimage_id.
After compass db recreate -y -k all eight suites in the affected set are
green. The lesson is written up in
Forty generated notify triggers
still emit the retired entity_ids payload, because the same confusion in
the other direction – committed SQL that really is stale – is a defect, and
the two look identical from a failing test.
5.4. The shape of the work
Six PRs, in order. PR 1 fixes the models and the one generator defect at source and regenerates nothing, because regeneration before those fixes would not compile. PR 2 points the suite at the ORE corpus and fixes the two parser defects it finds. PR 3 is the regeneration, the consumer migration and the dead-code deletion. PRs 4 and 5 are the oresmd native-identity cutover the abandoned sprint 25 story designed. PR 6 joins the registry.
5.5. What the corpus says about oresmd
ORE's shape table defines 43 series types. oresmd declares 29 of them, so
14 are unrepresented; measured against the committed corpus at
external/ore/examples/, those 14 are 48.8% of the key lines. The larger
problem is the other direction: 9 of the 29 declared types still cannot
express the keys ORE actually writes.
The shape table and the oresmd models contradicted each other, and the
shape table was right. ores_series_key_shapes_populate.sql states every
shape correctly – it was corrected from the corpus – and the oresmd
models were never brought into line. FRA and IMM_FRA were declared with an
index where ORE writes ccy/start/length, so all 2,273 of their corpus
lines projected nothing; MM's index-less form, 94% of its lines, was
rejected for want of an index the producer never wrote. All three now
project and round-trip.
What remains needs the identifier grammar extended rather than a shape corrected: BASIS_SWAP and the two cross-currency swaps carry two tenors and sometimes two currencies, ZERO carries a day count, the futures carry a contract month and an exchange, CORRELATION carries two operands on a strike surface, and CPR has no currency at all.
Two defects fell out of the same walk, and both are closed. The IR
6-segment path discarded the settlement segment and rebuilt it as a
hardcoded "2D", so the 1,338 corpus keys carrying 0D would have been
stored as 2D – PR 2 added settle to ir_market_data_identifier and
stopped rewriting it, and the segment now reaches the series qualifier as
the file wrote it. import_keeps_the_ir_swap_settlement_segment_the_file_carried
pins that for a 0D key and for one carrying an explicit settlement date,
which is the other thing the segment can be. The CSV tokeniser needed two
commas, so a line with exactly one aborted its whole file; PR 2 fixed that
too. What is left of the settlement question is the projection's "2D"
fallback, for an identifier built from something other than a key, and
there the proposed source – refdata's per-currency spot days – is the FX
convention rather than ORE's swap convention; see the analysis.
projects/ores.codegen/scripts/check_oresmd_ore_coverage.py makes the
coverage record permanent and rerunnable; --corpus adds the line counts.
It also enforces the test requirement: every modelled quote type carries a
projection test and a round-trip test, or a recorded reason. Seven of the
twenty-nine were one-sided when the check was written – six with no round
trip and one with no projection – which is how a projection that emits a
key its own inverse cannot read survives a green suite.
6. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
7. PRs
| PR | Title |
|---|---|
| #2185 | [marketdata] Make the models truthful and fix the junction gate |
| #2228 | [marketdata] Close the oresmd coverage gap and name every corpus key |
| #2243 | [marketdata] Regenerate, migrate the consumers, delete what generation supersedes |
| #2313 | [marketdata] Wire the import boundary through oresmd, and make its grammar honest |
| #2321 | [marketdata] Write the stored market data back out as ORE text |
| #2327 | [marketdata] Require a currency of a quote, not of a fixing |
8. Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|
9. Result
The checklist closes with 28 items Pass, one Partial and four Fail. The story's acceptance says every applicable item passes, so this task closes the story with those five recorded rather than met, and each names where it is tracked.
| Item | State | Disposition |
|---|---|---|
| M06 | Fail | Six hand-written protocol headers carry wire types no model covers. Deferred to a capture: six-marketdata-protocol-headers-have-no-model. |
| P04 | Fail | No operation model anywhere. The same six headers and the same generator gap as M06, deferred with it. |
| P01 | Fail | The canonical as_of read is absent from every marketdata list request. Re-scoped and deferred with H02 to backlog story F15A8480, "Generalize as-of and as-of-bucket queries across repositories". |
| W02 | Fail | The permission check passes vacuously because marketdata is not in its registry: 16 codes are checked, none is seeded as checked, and seven seeded marketdata:: codes are checked by no handler. Captured as CCB1D9D1, "Nothing gates a hand-written operation handler". |
| H01 | Partial | The composite root modeling/ores.marketdata.puml carries no diagram sentinel. A deliberate stop, not an oversight: it is a hand-authored data-model diagram for the oresmd identifier grammar, so the automated pass has nothing to contribute to a component whose root owns no code. |
What the four other tasks delivered: the checklist itself (28 items), the oresmd identity for every fixing index name (AB2922AA), the asset class lists declared once (E205D30E), and the oresmd URI as the series identity (4050BF6C). The follow-on work those units exposed is the story Make the oresmd URI grammar explicit and typed, which carries the grammar defects this story's identity cutover surfaced.