Task: Migrate the reference data documents

Move currency config, calendar adjustments, conventions, netting set definitions and credit simulation off the generated binding. Each family moves completely in one change: reader, writer, and the deletion of its mapper.

• Each family is read and written through the codec, with its mapper deleted in the same change.
• The corpus round-trip harness passes for each family's documents.
• No family has two live implementations at the end of the change.

One family at a time, each complete. A family moves reader, writer and mapper in one change, so no family ever has two live implementations. Calendar adjustments went first because it is the smallest and proves the shape every later migration reuses: its mapper is deleted, not retyped, and the projection that survives sits at the boundary beside the reading and the writing it serves. Currency config followed and needed two more codec helpers, because a currency's rounding type is an enumeration: the record holds an index and a projection speaks the schema's spelling.

The codec grew the vocabulary a projection needs. A projection receives the record the generated table describes and states which of its members it fills, because the writer writes only the fields whose presence bit is set. Four small functions join the codec: find_field, find_child_type, is_present and mark_present. A projection names a member the way the document does, so it does not repeat the schema or hardcode bit numbers. Because the lookup accepts an element or an attribute, the generator now refuses a type that spells both the same, so the ambiguity cannot arise silently.

A date list is two rows. The element that holds the dates and the Date row inside the record it points at are both rows, so a projection states both. The first attempt set only the outer one, and the dates vanished from the writing: the family's own round-trip test caught it before the change left the branch.

The importer reads text as well as files, so an export can be imported again without a temporary file, which is what the family's round-trip test does.

The netting set mapper had no reader. The ore-side mapper turned a netting set document into mapped_netting_sets, and only its own tests called it; the live netting set mapper is the refdata one on the database side. So the family needed no projection at all: its table was already written and proved, and the mappers and suites that duplicated it are gone.

Pricing engines moved as a projection, not a mapper. The family maps a document into analytics product and parameter rows, which is work no table can carry, so the class survives as pricing_engine_projection with the codec record as its input. It also owns reading and writing, so its two callers in the run store and the run configuration service ask it for a document rather than binding one themselves.

Two families share one projection helper. Currency config followed calendar adjustments, and its review found the table lookup copy-pasted in the importer and the exporter, and the writer's error message still naming the calendar table for a currency field. Both live in src/xml/projection.hpp now, beside the rule that a projection states every row it fills and throws when the schema no longer has it.

The remaining families, sized from a probe of the generator. Running the generator against each remaining schema says which need codec work and which need the generator extended first.

The generator reads a derivation. A type written as xs:complexContent extension now produces the base type's fields followed by its own, which is what flattening a derivation means for a record: the base's fields keep the offsets the base gives them. A regression test builds a base and a derived type and asserts the emitted record holds both, in that order. The run document generates with it. It is not in the regeneration wrapper yet, so nothing guards that table: it joins the wrapper in the change that migrates the family, because a table with no projection would leave the census with a supported family nothing reads.

One probe result was wrong. The table first said stress testing could not be generated because the generator lacked its shiftTypeEntry shape. It was the probe that was wrong: the type lives in sensitivity.xsd, and the family generates as soon as that schema is read. Only the run document still needs the generator extended, and that is xs:complexContent.

Credit simulation is not free. Its mapper is reached only by its own suites, which reads as dead code, but the configuration type is seeded and stored, so the family is a product concept with a test-only XML path. It is migrated, not deleted: the projection into the five analytics tables is the work.

conventions_mapper now takes the codec's records. read and write belong to it, and split_conventions and join_conventions keep their signatures, so the refdata side did not change. Enumerations are read as the schema spells them and collapsed to the canonical code through one table per enumeration, which a script derived from the old switch statements. Two codec functions were added for the one repeated enumerated list.

The export bytes changed, because the codec writes the document with its own serialiser. The recorded digests were taken again. A float the old binding held as float is now read as the document wrote it.

The generated domain, the xsdcpp scripts and the two regeneration patches went in the same change, with the 22 suites that only tested the generated binding and its golden dataset. The seed test lists simulation and sensitivity as types the codec does not carry.

The family is the largest in the corpus: 89 documents and a mapper of 3,310 lines. Its table is generated and proved over every one of those documents before any of the mapping moves, because the table is the part a schema decides, and proving it first separates "the schema reads differently than we thought" from "the mapping was transcribed wrongly".

It read differently in three places, and each was silent. A choice nested in a sequence, or whose alternatives are sequences, offered no members at all, so a document writing an equity volatility's VolatilityConfig or a commodity curve's SpotQuote did not bind. Two alternatives of one choice named the same element, which is one member in a flat record. And a member named after a type declared earlier in the same struct hid that type. The first two are generator work a test now covers. The third is a struct keyword on the one member that needs it, rather than a rename of a type whose name is fine everywhere else.

The census calls the family supported while its mapper is still the reader. That is the honest state of a two-step migration, and the evidence line in the census says so: the table reads and writes the document, and the projection replaces the mapper next.

The mapper was 3,310 lines and its input was the binding's generated types. The projection is the same mapping with the table's record as its input, so the schema's shape reaches it once instead of twice, and the mapper, its header and its suite are gone. Nothing about the mapping moved: an element it could not hold is still refused, and segments are still restored in the order the document wrote them.

Two conversions the binding used to make silently are now stated where they happen. The table holds every attribute as text, so a priority attribute is parsed and formatted; and an element typed xs:boolean is text as well, because it is not the ORE bool enumeration, so it is read as a flag and written as true or false. A strike factor is a schema decimal in a double column, and the boundary converts it.

The family's mapper was 1,230 lines and two suites, and nothing in the product called it: the importer, the run store and the run configuration service never name a stress document, and the file type table says the import cannot yet parse one. Its only callers were its own round-trip suites, which is the shape the netting set mapper had.

So the family did not need a projection. Its table is generated and proved over all 19 corpus documents, and the mapper and its suites are gone: the analytics stress test store keeps its own tests, and a later feature that imports a stress document will start from the table. The schema needs three files: stress.xsd, sensitivity.xsd for the shared element types, and ore_types.xsd for the enumerations. One name to know when reading the table: the SwaptionVolatility Shifts/Shift record is generated as shift_type because its element is named Shift, and it is not shift_type_entry.

The run document has the widest corpus of any family: 422 files under 235 names, six of which carry no ore prefix, so the value harness finds it by its root element. All of them bind, write and keep every value. The table is 269 lines across the two generated files; the mapper it will replace is the one the run store and the run configuration service call, and that projection is the next change.

What the corpus does not state is worth knowing: every document writes Setup and Analytics, most write Markets, and none writes Logging or Restore. Those two members are proved by the schema and the table, not by a document, and the projection that follows has no corpus fixture for them.

The projection is the mapper's rename table with the table's record as its input, and it owns reading and writing, so the run store and the run configuration service ask it for a document. The mapper, its header and its suite are gone.

Two things do not survive positionally, and the suite states both rather than comparing records blindly: the Setup parameters are written in a fixed order because the entity is a row of columns, and each analytic's active flag is written first because that is where ORE writes it. The suite compares Setup by name and value and each analytic by name and value, and it compares the analytics in order and the markets as a subtree.

The generator reads every shape the portfolio schema uses: a model group, a substitution member that writes its own type, a reference to a global element of an earlier schema file, an element with no type, a presence mask over more than 64 fields, and dependency-ordered types.

A composite trade holds a list of subTradeGroup entries whose SubTrade member is a componentSubTrade, which carries oreTradeData, which reaches the composite trade again. A record is a fixed layout and cannot hold that cycle, so the edge that closes it becomes an index row: the member is a std::vector<std::uint32_t> of indices beside a std::vector<T> arena in the same record. The arena belongs to the record that holds the indices, so copying the record copies its rows and no second object has to agree about their lifetime.

The binder appends the row to the parent's arena and pushes its index before it descends, so no row of the parent moves while a child frame is open. The emitter follows the indices in order. A row two indices share is written twice, which is the trade-off the database model makes. An index past the end of the arena, a row no index names, a second row where the schema allows one, and nesting deeper than the binder's 32 frames are all refused rather than dropped.

The table is generated from input.xsd with the nine schemas it reaches. The value harness binds, writes and reads back all 374 corpus portfolio documents, including the 16 that nest a composite, and no value changes.

Three rules the corpus found, each a codec rule now:

• A real is carried as an exact decimal. ORE writes 0.050000000000000003 and 7342.8000000000002 from a double, and no double reproduces that text, so the family states that every real is exact. The value a document states is the text it wrote.
• A number may sit on its own indented line. The number's lexical space allows the space around it and std::from_chars does not, so the reading trims a number's text before it parses it.
• The runs of one text are one value. A comment or a CDATA boundary splits an element's text, so the reading joins the runs and stores them once. A number split that way is the number its text states.
• A text element is not a number, whatever its name. Strike is a script variable in a scripted trade and a number in an equity digital option, so the exact decimal rule names reals only and leaves every text element as text. An element typed xs:simpleContent keeps its number as its own text, and an element that states no text leaves that number absent rather than zero.

The census now reports 13 supported families, 0 migrating and 7 unused.

What the table does not check yet. An xs:choice group is flattened, and each alternative is marked optional because the record has no way to state that one of them must appear. instruments.xsd makes DeliveryStart with DeliveryEnd, or PowerLoadProfileData, mandatory, and a portfolio document that states none of them binds. The values are not lost; the reading is looser than the schema. Teaching the table a required group is its own change.

The product still reads a portfolio through the old binding: importer.cpp, exporter.cpp, trade_mapper and one mapper per instrument family. The projection is the same mapping with the codec record as its input, so the schema's shape reaches the product once.

The shell owns the document, the envelope and the trade list. Each family owns the trade types it holds, and a family that does not handle a type returns nothing so the next one is asked. A projection names the elements it takes (the table is asked for the offsets), reads a money value as the exact decimal the table keeps and converts at the boundary, and refuses an element the mapping cannot hold rather than dropping it.

The scripted family came first because it is the smallest. The script's NPV is a name the schema requires and the instrument's flat table has no column for, so it rides in the parameters JSON the table does carry. Rates, FX, bonds, credit, equity, commodities and composites followed, and the importer and the exporter now ask the projection for the document and the trades. The nine mappers and their thirteen suites are gone, so the product never runs both paths.

Three things the port changes on purpose. A quantity the old binding read as float is now the document's exact decimal, so a stored amount no longer carries float noise. An FX forward or swap settlement is written back, which the old mapper read and dropped. A required element the trading domain holds no value for is stated at its structural minimum (one unit of quantity, an absent date, a zero initial price) rather than left out: the old binding tolerated the absence, and the document it produced did not bind.

The export is the part the old boundary never proved. Thirty-one of the 132 corpus documents used to export a document the reading refused, because a writer left a required element unstated. Every corpus document now exports and reads back, and the directory round trip asserts it.

Verified at the head of the portfolio projection branch, with the commands that prove each line.

What each remaining family needs. Portfolio's table is proved, and its projection replaces the importer, the exporter, the instrument mappers and trade_mapper next. Conventions and today's market are supported families whose legacy readers the importer still calls, and they leave with the binding.

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

todays_market_projection replaces todays_market_mapper and owns reading and writing, so the run store and the run configuration service ask it for a document. The twenty-four collections are four shapes, so one table of codecs states each collection once, and one table of binding attributes serves both directions. The old mapper held the same knowledge twice, once per direction.

The mapper's tests moved with it. The corpus walk now compares records, not written text, so an attribute written empty and one left out stay different. It finds 99 documents, not 98: simm_todaysmarket.xml has a root element the old walk's file-name prefix missed, and it round trips. The two structural suites that loaded the generated binding duplicated the value harness and are gone.

• A rename now fails instead of writing nothing. mark_present reports a name the table does not have, and the projection throws rather than emitting a document missing the field.
• The netting set family has a table and no reader. Its XML mapper was reached only by its own tests, so the mapper and those tests are gone and the family is covered by the binding, writing and round-trip suites that walk the corpus. A sprint-26 story records netting sets as reading and writing through the refdata store; that store is live, but the ORE XML path into it never had a caller, and this change makes that explicit.
• The reading is stricter than xsdcpp. The binder refuses an element or an attribute the schema does not name, and the importer throws. That is the codec's policy on a family with no xs:any, and it is the behaviour the tables were chosen for; a document ORE tolerates but the schema does not describe is a document to look at, not to import quietly.
• The boundary still throws. import_calendar_adjustments_text and the exporter report a malformed document with std::runtime_error, like every sibling in the importer and exporter. Converting the whole boundary to std::expected is one change for all families, not one per family.

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Every reference data document is on the codec, conventions included. The family census reports 13 supported, 0 migrating and 7 unused. The XML suites pass, and every portfolio in the example corpus exports a document that binds.

The generated domain is gone. No product file includes it, and the XML suites, the family census and the drift check pass.