Story: Model the ORE report configuration as entities
Table of Contents
This page documents a story in Sprint 26. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
1. Goal
A report definition's configuration is a set of strongly typed entities rather than a flat row of switches: every one of the seventeen ORE configuration document kinds is identified, mapped to an owning component, seeded as a configuration type, and round tripped, import and export, through the entities over the vendored corpus. A report type is a foreign key from the definition, seeded with a stable code, and it names the workflow it selects and the configuration types a run of that type requires.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent sprint | Sprint 26 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-05 |
On 2026-10-05, at the sprint 26 close, this story was reworded to the scope it delivered and closed DONE. The remainder — the round trips still missing, the report-kind classification, the run-input packaging, and the entity models the hand-written tables still need — is Complete the ORE report configuration, and its twenty-two tasks moved there.
3. Acceptance
- Every entity in the seventeen ORE configuration documents is identified and mapped to an owning component.
- A report definition reaches its configuration through typed entities and foreign keys, never through a flat table of switches or an XML or JSON blob.
- Every report type is seeded with a stable code, and each one names the workflow it selects and the configuration types a run of that type requires. The report definition reaches its type by foreign key, not by a text code.
- Every configuration kind whose entities are modelled round trips end to end over the vendored corpus: the run document, the stress test, the credit simulation, the curve configuration, the shared reference configuration, the conventions, the pricing engines and today's market.
4. Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Scaffold story: Model the ORE report configuration as entities | DONE | 2026-09-27 | 2026-10-04 | Story scaffolding rides this task: documents, sprint wiring, and the scaffold PR. Close it before merging that PR. |
| Design the ORE report configuration entity model | DONE | 2026-09-27 | 2026-10-04 | Initial task for: Model the ORE report configuration as entities |
| Model the run document and the analytic list | DONE | 2026-09-29 | 2026-09-29 | Typed entities for the ORE run document: the setup block, the market bindings, and the ordered analytic list with a lookup of the analytic types. |
| Model the curve configuration entities | DONE | 2026-10-01 | 2026-10-03 | The curve recipes ORE builds from, currently only partly held as ir_curve_bootstrap_config: yield, credit, FX, equity, commodity, inflation, volatility and correlation curves with their segments and quotes. |
| Import a run's configuration from an ORE input directory | DONE | 2026-10-04 | 2026-10-04 | Store a run document and the configuration documents it names against a report definition, and load them back as the files ORE reads, through one store per document kind. |
| Produce the engine's input from the entities | DONE | 2026-10-04 | 2026-10-04 | Build the ORE run document and the configuration documents it names from the entities, replacing the current package step that emits trades.msgpack and market_data.msgpack. |
| Import a run's configuration through a service operation and the shell | DONE | 2026-10-04 | 2026-10-05 | Only code can call the run store today; add a service operation and a shell command that import an ORE input directory against a report definition and export it back. |
| Implement the report configuration registry | DONE | 2026-09-28 | The reporting half as real tables: the definition, its slots, the configuration sets, the parameters and the value domains, with every discriminator a foreign key to a seeded lookup. | |
| Seed the configuration lookups | DONE | 2026-10-02 | 2026-10-02 | The domain vocabularies every configuration file discriminates on: configuration and report kinds, analytic types, value domains, shift families, shift types and shift schemes, risk classes, and the ORE element each maps to. |
| Build the round-trip harness | DONE | 2026-09-28 | 2026-09-28 | One test rig that takes an ORE configuration file of a given type, imports it into the entities, exports it back, and reports the diff, so each file type can be proven as soon as its mapper lands. |
| Round trip the ORE run document | DONE | 2026-10-01 | ore.xml end to end: the Setup parameters, the Markets bindings and the ordered analytic list as entities, with an import mapper, an export mapper and a round trip over every ore.xml in the corpus. | |
| Round trip the stress test configuration | DONE | 2026-09-29 | 2026-10-01 | stresstest.xml end to end: the named stress library, its scenarios, and the shift families each scenario applies, with both mappers and a round trip over every file in the corpus. |
| Round trip the credit simulation configuration | DONE | 2026-09-28 | creditsimulation.xml end to end: the entities that migrate, the transition matrices and their cells, and the run settings, with both mappers and a round trip over every file in the corpus. | |
| Round trip the curve configuration | DONE | 2026-10-03 | 2026-10-03 | curveconfig.xml end to end: yield, credit, FX, equity, commodity, inflation, volatility and correlation curves with their segments and quotes, with both mappers and a round trip over every file in the corpus. |
| Round trip the shared reference configuration | DONE | 2026-09-28 | 2026-09-28 | conventions, currencies, calendaradjustment and the other reference documents end to end, with both mappers and a round trip over every file in the corpus. |
| Bind the report types and their taxonomy | DONE | 2026-10-03 | 2026-10-03 | Make the report type a foreign key from the report definition, replace the generated sample codes with the real taxonomy, and record for each type the workflow it selects and the configuration types it requires. |
| Round trip the conventions configuration | DONE | 2026-09-28 | 2026-09-29 | conventions.xml carries twenty-six convention categories and the mapper models nine, skipping the rest silently. Fourteen unmodelled categories and some missing fields on the modelled ones stand between the corpus and a round trip. |
| Model the convention categories the mapper skips | DONE | 2026-09-29 | 2026-09-29 | Fourteen convention categories the corpus uses have no entity and no mapper, so sixty-three of the seventy-two conventions files cannot round trip. |
| Carry ORE's currency type on the refdata currency | DONE | 2026-09-28 | 2026-09-29 | The currency mapper translates ORE's free-text CurrencyType into the refdata monetary_nature, and Major and Minor both become fiat, so the export cannot recover which one the document held. Give the currency a column that carries ORE's own value. |
| Map the remaining stress shift families | DONE | 2026-10-01 | 2026-10-01 | Thirteen of the fourteen shift families the corpus uses have no reader, so no stress document round trips outright. Map them, and the ParShifts flags with them. |
| Bind the standard profiles to the configuration entities that carry none | DONE | 2026-10-01 | 2026-10-03 | Solid Dirac's shell-surface census found that analytics' three stress-test entities render no shell unit because their models carry no :profile: at all. A repo-wide census found eight entity models authored from this environment with no profile binding, across analytics, refdata and reporting. Each one hand-authors a feature combination instead of naming it, so codegen cannot guarantee a known-good shape and no profile-enabled facet fires. Bind each entity to the catalogued profile its shape matches, drop the redundant per-feature declarations the profile now supplies, and regenerate. Do not touch the profile-less models owned by other environments; report them instead. |
| Round trip the pricing engine configuration | DONE | 2026-10-01 | 2026-10-02 | pricingengine*.xml end to end: the engine definitions a report's valuation selects, each naming the model and engine it binds and the parameters it carries. The entities already existed in ores.analytics, so the work was the mapper and the proof rather than the model. All 125 corpus files round trip, the database case passes, and existing databases gain the position columns through a migration. The description the task was scaffolded with was wrong and named engine-to-currency assignments and grid and script settings that the schema does not have. |
| Round trip today's market configuration | DONE | 2026-10-01 | 2026-10-02 | todaysmarket*.xml end to end: an ordered set of 24 optional collection kinds — yield, discounting, index-forwarding, swap index, inflation, fx, equity, commodity, correlation, default and volatility curves and spots — each kind written once or more under its own id, plus Configuration blocks that select one collection of each kind by that id. Five tables, both mappers, all 98 corpus files and a database round trip. |
| Fix text escaping in the ORE bindings and the SQL codegen | DONE | 2026-10-02 | 2026-10-02 | The ORE XML binding decodes entities in attributes but not in element text, so an imported ampersand is stored as & and the export double-escapes it; the round-trip harness was changed to stop seeing this rather than fixed. The SQL codegen emits a foreign key error_message into a string literal unescaped, so an apostrophe aborts a database recreate. Fix both at their generators and restore the harness check. |
| Find why so much SQL is hand-written and generate what codegen can | DONE | 2026-10-02 | 2026-10-02 | Every component's create and drop aggregate is hand-curated, and 144 of the 754 create scripts carry no generated marker, so a new entity silently misses a recreate until someone lists it by hand. Measure what is hand-written and why, decide what codegen can own, and turn the wiring check that exists today into a gate. |
| Generate the hand-written RLS policies from model flags | DONE | 2026-10-04 | 2026-10-04 | Fourteen hand-written policy files hold 251 row-level security policies, about 129 of them plain tenant isolation on tables whose model exists but does not set the flag. A policy calls a function defined in iam, and dq, workspace, refdata and trading are created before iam, so codegen must write these policies to a late per-component file rather than inline. Prove each generated policy is identical to the hand-written one it replaces. |
| Link the pricing engine configuration to its report configuration | DONE | 2026-10-02 | 2026-10-02 | The credit simulation, stress test and today's market configurations each carry a configuration_id to the reporting configuration they are registered as, which is how a report reaches them. The pricing engine configuration does not, so a report cannot reach the engines it prices with. Add the nullable link, as the other three have it, with a migration for existing databases. |
| Seed the today's market collection kinds | DONE | 2026-10-03 | 2026-10-03 | Today's market collections and configuration bindings hold their collection kind as free text, and every entry row repeats the kind's key attribute. Seed a lookup of the twenty-four kinds with their ORE element and key attributes, make both collection columns refer to it, and take key_attribute off the entry table. |
| Seed the stress shift families and shift types | DONE | 2026-10-03 | 2026-10-03 | Stress test shifts hold their family and shift type as free text. Seed lookups of the families and shift types the stress schema defines, and make both columns refer to them, so the database refuses a kind ORE does not have. |
| Write floating point values to the database without rounding | DONE | 2026-10-03 | 2026-10-03 | sqlgen writes every float and double with std::to_string, which keeps six decimal places, so a curve tolerance of 1e-12 is stored as 0 and every double column is rounded to six decimals. Patch the sqlgen overlay port to write the shortest text that reads back as the same value. |
| Discuss the _config suffix for the curve configuration tables | DONE | 2026-10-03 | Talk through renaming the curve configuration tables so their names say they hold configuration, not curve data: yield_curve to yield_curve_config, fx_volatility to fx_volatility_config, and the rest. | |
| Rename the curve configuration tables with a _config suffix | DONE | 2026-10-03 | 2026-10-03 | Give every per-entry curve configuration table a _config suffix, so its name says it holds a curve's recipe, not its data; curve_definition keeps its name. |
| Remove the migration scripts | DONE | 2026-10-03 | 2026-10-03 | Delete the one-shot migration scripts under projects/ores.sql/migration and the policy that allowed them: the database is trashed and recreated, never migrated. |
| Scope the configuration documents to a party | DONE | 2026-10-03 | 2026-10-04 | Every ORE configuration document is per tenant and per party: party_id on every table, restrictive party RLS, keys and checks that let two parties hold the same names. Currencies, currency pairs and calendar adjustments stay per tenant. |
| Generate the schema fingerprint grant from its template | DONE | 2026-10-04 | 2026-10-04 | The grant that lets every service read the schema fingerprint was added to the generated grants file by hand, so the next regeneration deletes it and every service refuses to start; move it into the template. |
| Give each owning component its configuration document store | DONE | 2026-10-04 | 2026-10-04 | Move each configuration document's type and its store out of ores.ore into the component that owns its tables, so ores.ore maps XML without touching another component's repositories. |
| Expose the configuration document operations from each owning service | DONE | 2026-10-04 | 2026-10-05 | Give refdata, analytics and reporting NATS operations to save, get and delete their configuration documents, so other services reach them without touching their tables. |
| Refuse a second build in the same worktree | DONE | 2026-10-04 | 2026-10-04 | The build lock caps builds per host, not per worktree, so two compass builds of one worktree took both slots and raced in one build tree for over an hour; compass now refuses a build or test run while another holds the worktree. |
5. Decisions
- The plan of record is one task per ORE configuration document, each running the whole path for that document: entities, an import mapper, an export mapper, and a round trip over every file of that type in the vendored corpus. Four earlier tasks were shaped by concern rather than by document and now roll up into these, closing when their last file type lands: Model the run document and the analytic list into Round trip the ORE run document, Model the analytics configuration libraries into the seven analytics round trips, Model the curve configuration entities into Round trip the curve configuration, and Model the shared reference configuration into Round trip the shared reference configuration. The concern-shaped tasks stay open as the roll-up, so the story keeps a record of the design intent behind each slice.
- A file type is round tripped when the exported document equals the imported one as a parsed document, and the test names the first difference when they differ. The corpus is the evidence, not a hand-written fixture, except for the two types the corpus does not contain.
- The mappers follow the convention already in the tree: a class named <thing>_mapper with static map for ORE XML to our type and reverse for the other direction, beside the generated bindings in projects/ores.ore. There is no base class and no registry today, so the harness is what will enforce the convention. The existing xml::roundtrip directory walker covers four document kinds; the harness extends it rather than replacing it.
- simmcalibration.xsd is included by external/ore/xsd/input.xsd but SIMMCalibrationData appears zero times in the generated projects/ores.ore/core/include/ores.ore.core/domain/domain.hpp, so that document cannot be read or written at all today. The SIMM slice generates the binding first; the round trip proves it afterwards.
- The classification task was written against columns that do not exist. The
report definition carries
fsm_state_idandscheduler_job_id, and keepsschedule_expressionandconcurrency_policy;pre_processing,prepared_input_keyandpost_processingappear nowhere but an untracked web build artifact. Classification replaces the type code with a foreign key and the run columns with a classification, and it does not hunt for those three. - A reporting table declares a foreign key in a
* Foreign keyssection with:table:,:target_column:,:parent_entity:,:nullable:,:error_message:and:list_by:, which generates validation and indexes; it does not use the:references:property that one compute model uses. Seeevidence/reporting_model_recon.md. - A round trip covers a document kind, so the corpus is the set of files whose
root element is that kind, found by glob. Counting files named exactly
<kind>.xmlundercounts: the run document is 416 files, not 82, and credit simulation is 14, not 2, because the corpus holds variants such asore_histvar.xmlandsimm_calibration.xml. Seeevidence/round_trip_corpus.md. - Three documents cannot be round tripped as they stand.
simmcalibration.xsdandbaselTrafficLightconfig.xsddeclare global elements that appear zero times in the generateddomain.hpp, so no binding exists for them;historicalreturnconfig.xsddeclares no global element at all and is a section of the documents that reference it, not a file type. The two with a root element generate their binding first. - The existing binding round trips do not prove the content of every document. In `creditsimulation.xml` the transition matrix is carried as one `Data` element whose text is a whitespace or comma separated grid, with optional `t0` and `t1` attributes, and the existing `xml_creditsimulation_roundtrip_tests.cpp` deliberately does not compare `Data`. A test can therefore pass while the matrix is lost. Our round trip compares the grid cell by cell and names the first difference.
- A passing round trip can hide an axis it never measured. The credit simulation label comparison ran over an empty vector on both sides, because the binding does not hand the mapper the XML comment the labels live in: `parse_credit_simulation_grid` returned 64 values and no labels, and the loop that compared them executed zero times. Fourteen files passed a check that never ran, and it was reported as proof. The scale is fixed and is now in the type, so both directions take the ratings from one ordered constant and the comparison has a real expected value. An assertion over an empty vector is not an assertion, and a test that cannot fail is worse than no test.
- Where a configuration's entities live. ORE's own
Analyticsblock decides it. If ORE names the document in that block it belongs toores.analytics; if the run or an analytic merely references it, it belongs to the component that owns the concept, which isores.refdatafor curves, conventions, calendars, currencies and tenors, andores.reportingfor the run document and the registry.ores.oreholds the mappers and no state. A configuration is a header row inreporting.configurationsand its content is in the owning component, joined byconfiguration_id. - The harness takes a per-kind comparison, because parsed equality is not
text equality. The binding holds some numeric attributes as text and the
mapper writes them back in its own spelling: ORE's
t0"0.0"= and ourt0"0"= are the same bound, and the netting set list is indented differently although its codes are the same. A generic text comparison fails a faithful round trip; a rule that ignores text hides a lost value. So a kind supplies its comparison beside its mappers and the harness owns the walk, the count and the failure list. The credit simulation comparison moved out of its test intocredit_simulation_diff.cppfor this reason, so the test and the harness share one implementation rather than drifting apart. - A comparison may state a normalisation, but only one that carries the same information. Calendar adjustments are the case: ORE writes a date list it has no dates for as an empty element and the mapper omits it, so the kind declares that an empty list may be written either way. The rule is narrow by construction, because a list with dates in it is compared date by date and cannot slip past it. A comparison may not be widened to accept a lost value: where the difference is a field the entities cannot hold, as with ORE's currency type, the model is wrong and the model is what changes.
- The harness walks a corpus by file prefix, which is what the census globs.
A kind's files are the ones whose stem starts with the kind's prefix, so
creditsimulation*finds the 14 credit simulation documents includingcreditsimulation3_ts.xml. Counting files named exactly after the kind undercounts, which the census already recorded. - A mapper that skips must count what it skips.
conventions_mapperignored seventeen of the document's twenty-six categories and said nothing, so seventy-two files passed a round-trip test that compared element counts while losing content. It now fillsmapped_conventions::unmodelledand warns per category. A gap that is counted is one a caller can act on; a gap that is silent is indistinguishable from a document with nothing in it. - A normalisation the mapper performs on purpose is stated, not fixed.
Conventions come back with
truewritten asTrueandA365asA365F, because the mapper collapses booleans and enum aliases to the canonical codes the refdata columns hold. Two spellings of one boolean, and two aliases of one day counter, carry the same information, so the kind's comparison states them. The rule this story already holds to applies unchanged: what may be stated is a difference that carries the same information, and a dropped element carries less than the document did. - A vendor's token gets its own column. ORE's
CurrencyTypewas translated into the refdatamonetary_nature, and the translation could not be undone:MajorandMinorboth becamefiat, so the export could never recover which one the document held. The two are different axes, since a nature says what a currency is and ORE's type says how important it is, andmarket_tierwas not a home for it either because its lookup holdsg10andemerging. The currency now carries ORE's token verbatim inore_currency_type, which is nullable because ORE declares the element as a free string and most documents set none, and it derives the nature from it. The rule generalises: where a document's field is the vendor's vocabulary and not ours, it is stored as the vendor wrote it, and our classification of it is a separate derived column. - A round trip is compared after a save and load cycle. A document that cannot survive its own export and re-import is not a round trip, and the cycle is what proves the binding decodes the text it escapes. PR #2500 dropped the cycle to hide a binding defect; it is back, with the defect fixed.
- The ORE parser is fixed in
xsdcpp, not in the bindings. The generator embedssrc/XmlParser.cppverbatim intodomain.cpp, so a parser fix made only in ORE Studio would be lost at the next regeneration. Until the unexplainedtradeandcomponentSubTradedrift in a full regeneration is resolved, the bindings take only the parser hunks, byte-identical to the generated ones. - A file's origin is measured by rendering, not read from its comment.
census_sql_origin.pyrenders every catalogue component and labels a SQL file codegen output exactly when the render contains it. Generated files also carry the "do not edit" comment, drop scripts included, but a comment survives the deletion of its generator's input, so it is a hint for a reader, not a count. - Every SQL file is wired or named as an exception.
WIRE_001checks every file undercreate,dropandpopulate, reached from the tree roots and the bootstrap scripts, and CI fails on it. A file a person or another tool runs directly goes invalidation_ignore.txtwith what runs it. - A report type selects the workflow and names the run's required configuration. The workflow is a column on the type, and the required configuration types are a junction, both read under the system tenant where the lookups are seeded. The trigger refuses a definition that binds fewer, so a run that cannot finish never creates an instance. Analytic-specific configuration belongs on the analytic types, not the report type.
- Soft foreign keys impose no creation order. An insert trigger resolves its parent table when it runs, so generated aggregates need only a table before its trigger and hand files that name tables after the tables.
- World data is not party configuration. World data (currencies,
calendars, index and currency pair conventions, market quotes) is one view
of the world that every party shares, so it has no
party_id; it is per tenant until the synthetic data rewrite settles its scope. Party configuration (run documents, pricing engines, curve configurations, today's market, stress tests, instrument conventions) hasparty_idand a restrictive party RLS policy, set from the JWT bystamp. Synthetic data writes only world data. See World data versus party configuration. - A key column can be wider in SQL than in the API.
:sql_key: trueputs a column in the primary key, unique indexes, exclusion constraint and versioning triggers but not in the repository or protocol key, so a party supplied by the session lets two parties hold the same convention id. - A run's configuration moves through one store per document kind.
ores.ore.core/storewrites and reads each mapped document, and the run store imports an ORE input directory against a report definition and exports it back as the files the run document names. A binding to conventions means the conventions the session sees, since conventions have no header. - Reporting lookups are read in the system tenant. Configuration types,
analytic types and parameter definitions are seeded once, for the system
tenant, so every foreign key to them carries
:use_system_tenant:. - No service reads or writes another component's tables. The run store
that Import a run's configuration from an ORE input directory put in
ores.ore.corewrote the reporting, analytics and refdata tables directly, and the operator ruled that an architectural error. Each owner stores its own configuration documents and exposes them as NATS operations;ores.oremaps ORE XML to and from those documents and orchestrates the owners, with a workflow saga for the import. See No service reads or writes another component's tables.