Story: Decide the XML strategy for ORE documents

Table of Contents

This page documents a story in Sprint 28. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

1. Goal

Decide what the layer between an ORE XML document and the ORE Studio model is, and settle it with measurements rather than preference.

The bond review found that a money value cannot be recovered exactly from the generated binding wherever ORE types the element as a bare xs:float. That was one symptom. The layer has two more problems of the same origin: it builds an object graph where the guidelines say the document tree is transient, and it splits one mapping across 122,000 generated lines and 800 KB of hand-written mapper that cannot be checked against each other.

The story is done when the route is chosen, the evidence is on the page, and the work that follows is broken into tasks.

2. Status

Field Value
State DONE
Parent sprint Sprint 28
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-20

3. Acceptance

  • One investigation page states the design, the alternatives and the decision.
  • One prototype demonstrates the design on a real ORE document and reports time, allocations and fidelity against the current path.
  • The prototype's output agrees record for record with the production path on the document it covers.
  • The follow-on work is expressed as tasks with sizes, or the route is declined with the reason recorded.
  • Numeric accuracy is stated as an invariant, not a hope: money is a decimal taken from the document's digits, continuous quantities stay binary, and the writer emits a form that reads back to the same value.
  • Round trip is defined as the values surviving and the output validating against input.xsd, not as the document being reproduced.
  • Every corpus root element is either on a table and proved by the harness, or named by the census as still migrating because product code reads it, or deleted because nothing does. No family is silently left behind.
  • Every XML document in the repository parses with the new scanner, and the check that proves it fails the build when one does not.
  • The write direction is specified, including the one place where shortest round-trip formatting is wrong.

4. Tasks

Task State Start End Description
Prototype the descriptor document codec and measure it DONE 2026-10-17 2026-10-17 Build a throwaway codec that walks generated descriptor tables instead of a generated DOM, then measure time, allocations and fidelity against the current xsdcpp path.
Gate the scanner on the XML corpus DONE 2026-10-17 2026-10-17 Run the scanner over every XML document in the repository on every check, and fail on any rejection. 3,971 documents and 44.8 MB today, including the FpML and FIX trees, which are far outside the ORE subset.
Generalise the codec for nested and repeated types DONE 2026-10-17 2026-10-17 The probe maps one flat record family. Extend the engine and the generator model to nested complex types, repeated children, attributes on leaf elements, and xs:any, and state the policy for content the schema does not name.
Generate tables for every document family DONE 2026-10-17 2026-10-19 Read all twenty-eight ORE XSDs and emit a table per complex type, with a drift check that fails when a table and its schema disagree. This is the replacement for the generated domain, so it must be reproducible and reviewable.
Write the exporter from the same tables DONE 2026-10-17 2026-10-17 The write direction need not mirror the read direction. Emit in schema order, write only present fields, format decimals as fixed point rather than shortest round trip, escape only where needed, and build one buffer per record range.
Migrate the reference data documents DONE 2026-10-17 2026-10-19 Currency config, calendar adjustments, conventions, netting set definitions and credit simulation. Each moves its importer and exporter onto the codec and deletes its mapper in the same change.
Migrate the portfolio documents DONE   2026-10-19 Trades, envelopes and the instrument families. The largest mapper group and the one the bond work touched, so it lands after the reference data proves the shape.
Migrate the market and configuration documents DONE   2026-10-19 Curve configuration, today's market, pricing engines, stress testing and the ORE run document.
Remove xsdcpp DONE   2026-10-19 Delete the generated domain, the sibling-checkout wiring, the two post-regeneration patches and the knowledge page. Last, because nothing may depend on the generated binding by then.
Stop formatting a log line per imported record DONE   2026-10-19 The import path formats a full JSON record for every mapped entity, and it is the largest single cost in the current importer. Guard the trace lines so an off-level log costs nothing.
Prove the round trip preserves the data and validates DONE 2026-10-17 2026-10-17 Round trip means the values survive and the output validates, not that the document is reproduced. Build the harness: read the corpus, write it, validate against input.xsd, read it back, and compare every field including the exact decimals. Fail on any element the model did not claim.
Map the configuration documents that have no mapper ABANDONED     Ibor fallback, reference data, script library, counterparty information and collateral balances. No mapper exists today, so these are new domain models on the codec rather than migrations.
Map the analytics documents that have no mapper ABANDONED     Simulation, sensitivity analysis, SIMM calibration and the cross asset model. The largest of the unmapped families and the one that exercises the most of the schema.
Build the faithful document layer DONE   2026-10-17 Put the scanner in product code with the transient document model and the writer, and prove over the corpus that every document reads, writes back, compares equal and still validates against input.xsd. This layer needs no schema, so it lands before the schema tables.
Delete the document families nothing reads DONE 2026-10-17 2026-10-17 Seven of the twenty corpus root elements have no product consumer. Do not map them; delete their tests and their dead mapper, and record the census that says which families those are.
Cut the vector growth in the portfolio read DONE 2026-10-19 2026-10-19 The portfolio read spends a third of its instructions growing the trade vector and moving large trade records. Measure, then reserve or avoid the move.
Store each trade kind in its own column DONE 2026-10-20 2026-10-20 The generated trade record holds one inline member per instrument kind, so it is 346,624 bytes and every new trade zeroes all of it. That zero fill is 33 percent of the portfolio read test. The kinds now sit in columns: the trade keeps its common fields and one index, the presence bit says which kind, and the holder of the trade owns one vector of rows per kind type. A trade is 432 bytes and the test costs 39 percent fewer instructions. No variant is used.
Profile the portfolio read again and measure peak memory DONE 2026-10-20 2026-10-20 After the trade columns the zero fill is 3.6 percent. Find the next cost of the portfolio read, fix what is cheap, and measure the peak memory of a large import.
Remove stable_arena and close the XML story DONE 2026-10-20 2026-10-20 Nothing in the generator uses stable_arena since trade kinds moved to columns. Delete it, and close the XML story with what it delivered.
Cut the rate instrument and leg read cost DONE 2026-10-21 2026-10-21 read_rate_instrument costs about 59,000 instructions per trade and the leg append about 13,000 per leg. Profile both by line, remove the largest avoidable costs, and measure.
Remove the last xsdcpp references DONE 2026-10-21 2026-10-21 Comments, a memory, a runbook step, a knowledge page, an analysis and a plan still name xsdcpp or its scripts. Rewrite them to the current state or delete them.

5. Decisions

  • A family nobody reads is deleted, not mapped. Seven of the twenty corpus root elements have no product consumer, so a table for them would prove a capability no feature has. They are deleted with xsdcpp, and a table is added if a feature needs one. The census in build/scripts/check_xml_family_coverage.py names the state of every family and fails when the tree disagrees with it.
  • The document tree is transient. The guidelines place ORE documents at the boundary, outside the three regimes, and name the entity representation as the review target. The design follows that rather than the usual shape of an XML library.
  • Three routes were weighed; route C is recommended. Keep and extend xsdcpp, adopt a document library, or own a scanner driven by tables generated from the schema. The investigation carries the numbers.
  • The mapping becomes data. A table can be checked against the XSD by constructing it. The mechanical part of the mapping moves into the table, and the semantic part stays a short function.

6. Out of scope

  • Extending xsdcpp. Route A is described and rejected, not scheduled.
  • Reproducing a document byte for byte: comments, indentation, attribute order and the original spelling of a number are out of contract.
  • A general XML library: XPath, run-time schema validation, a DOM.
  • The drift on trade and componentSubTrade. Route C does not touch the generated binding, so the drift stops blocking anything.
  • Removing xsdcpp before every root element has a codec. It is the last task, not an early one.

Emacs 29.3 (Org mode 9.6.15)