Story: Make ores.ore the only owner of ORE formats

Table of Contents

This page documents a story in Sprint 29. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

1. Goal

ores.ore converts between ORE formats and our domain types and does nothing else. Every ORE-defined format is read and written in one place, the component's dependencies are api contracts rather than other components' cores, and a build check stops the boundary from drifting again.

2. Status

Field Value
State DONE
Parent sprint Sprint 29
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-28

3. Acceptance

  • ores.ore.core links no other component's core or service library; its dependencies are api contracts, logging, platform, utility, pugixml and Boost.
  • The codec performs no database work: it opens no session, calls no repository and writes no table.
  • The setup parameter that names each ORE input file is declared in ores.ore and checked against external/ore/xsd, not read from a configuration table.
  • One parser reads the engine's log, and it reads both log.txt and log_structured.json.
  • A build check fails when a component's core library links another component's core or service library.
  • The ORE document kinds the codec recognises, and the formats it converts in each direction, are recorded in one place and agree with external/ore/xsd.

4. Tasks

Task State Start End Description
Remove the database store from the ORE codec DONE 2026-10-28 2026-10-28 The ORE codec's run store writes reporting, refdata and analytics tables through their repositories and services, and only its own tests call it. Remove the import and export, keep the package helpers, and drop the core libraries the codec no longer needs.
Move the conventions bundle into refdata's contract DONE 2026-10-28 2026-10-28 conventions_package.hpp includes ores.refdata.core for conventions_bundle_row, a three-field payload type. That one type is why the ORE codec depends on refdata's service layer. Move the bundle types to refdata.api, where the stage and commit protocol that carries them lives.
Declare the ORE input files in ores.ore DONE 2026-10-28 2026-10-28 The run configuration import learns that the conventions document is named by the conventionsFile setup parameter by reading reporting's file_types.ore_parameter column, a seed generated from external/ore. A fact of ore.xsd belongs in ores.ore.
Read the engine log with one parser DONE   2026-10-28 ores.ore carries two log parsers. ore_log_parser returns a normalised line and has one client, the compute node's log publisher; engine_log reads both log.txt and log_structured.json. Two readers of one format disagree eventually.
Edit the run document through the projection DONE   2026-10-29 The package handler finds the as-of date by searching for the text name="asofDate">, and finds the latest market data date by looking for a tab in column nine. Both formats have an owner in ores.ore, and neither edit goes through it.
Delete reporting's copy of the ORE log parser DONE   2026-10-28 reporting parses ORE's log.txt itself in engine_error_log, and cannot link ores.ore while the codec depends on reporting. With the codec clean, reporting keeps the canonical row type and deletes the grammar.
Move the cross-component tests out of the ORE codec DONE   2026-10-29 Eight test files in the codec's suite opened a database, and three of them read reporting's tables rather than the codec: the configuration type seed, reporting's configuration repository, and the run store this story deleted.
Fail a build that links another component's core DONE   2026-10-29 Nothing stops a component's core from linking another component's core or service, which is how the ORE codec reached into reporting. Encode the rule in the component model and check it in CI.
Classify every ORE document and record what we convert DONE     document_kind knows six of the thirteen generated families, so the directory scanner cannot name a run document, a curve configuration or a today's market. Five families have a binding and reach nothing.
Scaffold story: Make ores.ore the only owner of ORE formats DONE 2026-10-27 2026-10-28 Story scaffolding rides this task: documents, sprint wiring, and the scaffold PR. Close it before merging that PR.

5. Notes

The audit that scoped the story measured six dependencies the codec does not justify. The Status table claims the run store drops four of them, which is the table below.

# Dependency Kind Why it is there Dropped by
1 ores.database public the run store's session context the run store
2 ores.analytics.core private the run store's document services the run store
3 ores.reporting.core private the run store's repositories the run store
4 ores.refdata.core public the conventions bundle type, and the run store's curve service the run store and the bundle move
5 ores.nats public nothing the codec uses; the generated core profile adds it the profile, or a per-component override
6 ores.dq.api private four change-reason constants in the planner the planner taking them as a parameter

The api contracts the codec keeps are justified by the mapping it performs: ores.refdata.api, ores.trading.api and ores.analytics.api. ores.reporting.api stays for the run document projection, which is the conversion the component exists to perform.

6. Decisions

  • 2026-10-27: The story is scoped from an audit of the codec's dependencies. The codec itself is clean: xml/generated is produced from external/ore/xsd by our own generator, and no XML parser exists outside the component. Every finding sits above the codec — the run store, two public headers, and one test suite.
  • 2026-10-27: The run store's import_run and export_run are deleted rather than re-homed. The live import is run_configuration_operations, which reaches each owner over NATS with a delegated token. The database path has no production caller, and only its own tests keep it alive. The package helpers stay, because an ORE input directory and its layout are ORE artefacts.
  • 2026-10-27: reporting.api stays a codec dependency. Mapping ore.xml into our run document is the conversion this component exists to perform, and the reporting contract is a leaf. What changes is the kind of edge that remains: no core, no service.
  • 2026-10-27: The XML binding generator is not revisited. xsdcpp is removed and the generator is ours. Its missing drift check is recorded out of scope so that it is not lost.
  • 2026-10-27: The state before the work is measured, not assumed. libores.ore.core.so declares 22 ORES libraries, including reporting.core, analytics.core, refdata.core, service and database. ores.compute.wrapper links the codec, so the compute node process loads the reporting service layer to parse a log and marshal a CSV.

7. Out of scope

  • A drift check for the XML binding generator. regenerate_xml_bindings.sh supports XML_BINDINGS_OUT for one, but no test and no CI job runs it, so the committed tables can disagree with external/ore/xsd. The component overview claims the check exists.
  • The conventions import and export redesign, which has a story of its own.
  • The oresmd:// URI grammar, hand-built in ores.ore.service, ores.iam.core, ores.synthetic.api and the shell. It is our format, not ORE's.

8. Result

ores.ore owns every ORE format. The codec reads and writes ORE documents through one generated binding, holds no database store, and states in one table what the tree does with each of the thirteen families the generator emits. Its declared libraries are down from 22 to 18, its suite opens a session only for the seed check, and a gate fails a build whose core links another component's core.

The work is nine tasks: the story scaffold, the store removal, the conventions bundle moved into refdata's contract, the ORE input files declared in the codec, one reader for the engine's log, reporting's copy of that grammar deleted, the run document edited only through its projection, the database tests moved to the suites that own a database, and every document family classified.

Emacs 29.3 (Org mode 9.6.15)