Task: Rework the ores.ore bond mapping to the relational model

Wave 1.5 left the mapper in a deliberate halfway state. One file (bond_instrument_mapper) carries forward and reverse for seven of the ten product codes. It targets the generated rows already. The container it returns (bond_instrument_data) is a handcrafted carrier: the instrument header, the issue row, one engaged fact row, and a document remainder (option_expiry_date). Wave 1.5 fixed the fidelity rule at "reverse emits exactly what it emits today", so the flattening survived the reshape. This task delivers the depth.

The ORE schema (external/ore/xsd/instruments.xsd) declares nine bond product structures: bondData, forwardBondData, callableBondData, convertibleBondData, bondOptionData, bondTRSData, bondRepoData, bondFutureData and ascotData. The schema declares no positionData. BondPosition therefore has no product document in the XML at all.

Nine of the ten codes are importable. Two of the nine have no mapper today: BondFuture and Ascot. Both have a fact table waiting (ores_trading_bond_futures_tbl with its full contract column set, ores_trading_ascots_tbl with ascot_option_type), so both gain a mapper pair. BondPosition is the recorded uncovered product: no product structure in the XML, no fact table in the family, and the deliverable already records that it has no mapper. Its absence is a decision with evidence, not an omission, and the suite asserts it.

Five items flatten today. Three have a destination row. Two do not. The difference decides the wave shape.

• The call schedule and the conversion terms have destinations now. bond_issue_call_dates is keyed (issue_id, sequence_number) and carries call_date; bond_issue_conversion_targets carries underlying_id and conversion_ratio. Both land as child rows. The XSD expresses each as a scheduleData plus a styles list, and scheduleData is a choice: an explicit Dates/Date list, or a Rules block. An explicit date list maps to a row per date. A rule-based schedule does not, because expanding a tenor against a calendar is schedule generation, not mapping.
• The full exercise date list, the funding leg's schedule and the repo leg's dates have no destination in the nine tables. This is the recorded wave 1.3 scope limit, not an oversight. The shared instrument-keyed schedule and underlyings tables belong to the parent story and are not part of this family. A bond-local schedule table would create the per-family duplication the program exists to remove, and the parent story would have to generalise or delete it on arrival.

The plan splits the five. The child rows land now. The items with no row stop flattening at the mapping boundary: the container carries the whole structure the document holds (every exercise date, the funding schedule, the repo dates), reverse re-emits it whole, and the residue is recorded against the parent story's shared tables. What the tables cannot store, the document keeps.

The acceptance names both. They are different kinds of defect. The corruption (forward_bond_repo overwriting the issue's real tenor with the leg type) is already gone as a mechanical consequence of the shared issue row. What this task owes is the test that asserts a repo product's issue keeps its own tenor while the repo row carries the leg type. The TRS hardcoding is live code: forward_bond_trs writes "TotalReturn" unconditionally, so a PriceReturn document imports as TotalReturn. The mapper reads the return side off the document and asserts both directions.

Each wave ends green and is a commit.

• 2.1, the survey. The per-product XSD field inventory, the ten-code coverage decision, and the flattening inventory, into Notes. Checks: docs class.
• 2.2, the issue side. A bond_issue mapper and the find-or-create boundary. The container mints a fresh issue per trade today, with a comment that finding an existing issue is the database boundary of this task. The assembler gains an explicit resolver seam, so "found or created" is a real lookup at the boundary and a mint on the miss. The DB path supplies the lookup; unit tests supply a stub. The two child mappers (call dates, conversion targets) land here, with the issue they hang off.
• 2.3, the fact mappers and the assembler. One mapper pair per fact entity over the five products, the two missing products (BondFuture, Ascot) added, the container grown to carry them and the per-product remainder, and the assembler that composes one document into issue, instrument, fact and children and joins them back.
• 2.4, the suites. The mapper and golden roundtrip suites extended for the new products and for the assertions the acceptance names (the repair, the hardcoding, the whole exercise date list, the child rows), plus the coverage assertion that names BondPosition.
• 2.5, validation. Full build and ctest, the ores.ore roundtrip suite, and the file-level check through ores.cli (exporter::roundtrip) with trades_mapped counting the covered products. Checks: code class, full build and ctest, plus the ores.ore roundtrip suite.

No new verb surface. The DB-bound import path stays workflow.v1.ore.import with its existing drivers, and the resolver seam is internal to the mapper. The container's full rework (one view per product, the wire protocol, the qt form) belongs to unit 3, which the parent plan sequences over the container grammar work. This task grows the container only as far as the mapper needs to carry what it maps. The shared schedule and underlyings tables of the parent story remain the destination for the three residues.

The assembler gained the find-or-create seam the plan named. All seven forward entry points take a bond_issue_lookup, a function that answers one question: does this security identifier already have an issue row? A hit adopts the row the security already has. A miss mints a fresh issue identifier. The mapper stays pure, because the lookup is a parameter and not a database handle. The import path supplies the real lookup, and the unit tests supply a stub.

The two child mappers landed with the issue they hang off. map_call_dates writes one bond_issue_call_dates row per explicit date in the Dates choice, numbered from one. map_conversion_targets writes one bond_issue_conversion_targets row per positive ratio, and carries the underlying name. Both reverse functions rebuild the document structure the rows came from.

The rule-based form of each schedule stays in the document. An explicit date list maps a row per date; a Rules block does not map, because expanding a tenor needs a calendar. The container carries the structure whole, and the parent story's shared schedule tables are the recorded destination.

One mapper pair per fact entity now sits in the assembler, and the container grew to carry what the nine tables cannot store. The wave also added the two products the earlier waves left out, BondFuture and Ascot.

The container gained the future and ascot fact rows, option_exercise_dates in place of the single option_expiry_date, trs_price_type, and three bond_leg_data members: the TRS funding leg, the repo leg and the ascot reference swap leg. A bond_leg_data holds the leg's payer flag and the schedule the leg's dates come from.

The two legs of the round trip are now complete for every product.

• map_schedule and reverse_schedule carry a scheduleData whole: both the Rules blocks and the explicit Dates lists, with every member of both.
• map_leg and reverse_leg carry a leg's payer flag and schedule.
• map_exercise_dates and reverse_exercise_dates carry the whole exercise date list, not one date.
• parse_code reads an enumeration spelling back. The generated domain exports to_string for every enumeration but no parse, and the generated toNumber helper is file-local to domain.cpp. The scan over to_string keeps one source of truth: a spelling the schema adds parses with no table here to keep aligned against.
• number_of, count_of and flag_of read the fields the schema states as strings. A value the parser cannot read leaves the column at its default rather than failing the whole import.

Every TradeType in the examples tree now has a mapper. A sweep of all 115 oreTradeType enumerators against the dispatchers finds 45 with no mapper, and no example document states any of them, so Ascot was the last uncovered document.

That breaks import_portfolio_with_context_unmapped_type_is_monostate, which used Cash_Ascot.xml as its unmapped fixture. The guard still earns its place, so the test now builds the trade the tree cannot supply: a BondPosition, a sub-trade type that no document states on its own, checked straight against trade_mapper::map_instrument.

The acceptance names two defects. Both are gone.

• The coupon frequency corruption in forward_bond_repo. The old mapper wrote the repo leg's schedule tenor into the issue's coupon_frequency_code column, because no column held a leg schedule. The leg's schedule now rides the container, and the issue column keeps the bond's own frequency.
• The hardcoded TRS values. reverse_bond_trs wrote PriceType as Dirty and both legs wrote Payer as false, whatever the document said. All three are gone: the price type and the payer flags are carried and written back.

Dropped fields remain dropped, and they are pre-existing scope limits rather than new losses. The totalReturnData fields other than PriceType, and the legs' currency, notional and day counter, were never mapped in either direction. Wave 2.4 records them as the remainder the parent story's tables take.

Two normalisations are worth stating, because they change the bytes and not the meaning. A flag the document states as false and a flag the document omits carry the same value, so a false flag exports as an omitted element. A boolean re-exports in the canonical spelling (Y, N, true), so a document that spells it TRUE round trips value-equal and not spelling-equal.

The wave 2.1 coverage table named Hybrid_GenericTRS_with_BondForward_Futures.xml as the BondFuture vehicle. It is not one. A sweep of the whole examples tree found bondFutureData in zero documents; that file holds a TotalReturnSwap over a ForwardBond sub-trade. The schema still declares both BondFutureData (instruments.xsd line 48) and the BondFuture trade type (line 1381), and only ORE-Python/ExampleScripts/portfolio2.py mentions the type anywhere in the tree.

The mapper is therefore written against the schema and unexercised by any example document. Wave 2.4 authors the fixture, as it does for the call dates and the conversion terms.

instrument.issue_id is NOT NULL, and bondFutureData carries no bondData. A future therefore mints an empty issue row, and the find-or-create lookup can never match one: there is no security identifier in the document to look up. The row is a placeholder that satisfies the foreign key. A future that later needs issue terms needs a schema field first.

option_exercise_dates is a list, and the container is right. The qt form has one QDateEdit, so writeUiToInstrument stages a single-element list and the load path shows front(). A document with three exercise dates round trips through the mapper intact and shows one date in the form. The reworked form is unit 3.

The trs org tells the reworked mapper to read return_type from TotalReturnData.PriceType. The evidence contradicts that instruction.

• The column check accepts two values. projects/ores.sql/create/trading/trading_bond_trs_create.sql line 59 requires return_type to be TotalReturn or PriceReturn.
• PriceType carries a different pair of values. The ORE User Guide (Docs/UserGuide/tradedata/bondTotalReturnSwap.tex in the upstream ORE repository) gives the allowable values as Dirty or Clean, and states that the total return comprises the coupon, redemption and amortisation payments of the bond plus a compensation payment for the change of the clean bond value.
• Every BondTRS document in the examples tree carries Dirty. So does every BondOption document. Dirty names a price-quoting convention, not a return type.

An ORE BondTRS is a total return swap by construction, so no field in the document selects a return type. Two consequences follow.

• Forward: return_type becomes a documented model default of TotalReturn, with the User Guide as the evidence. The value is correct for every document the schema can express.
• Reverse: the real hardcoded value is PriceType, which reverse_bond_trs writes as Dirty today. PriceType rides the container remainder, so a document that carries Clean round trips unchanged.

The acceptance says the hardcoded TRS values disappear. They do, on the export side, where the value is real document data. The import side keeps a constant that the schema justifies, and the note here is the record of why.

Three suites carry the wave's assertions. xml_bond_fact_mapper_roundtrip_tests is new and holds the per-product facts: the repo leg and the coupon frequency repair, the TRS price type and funding schedule, the future's seventeen columns and its delivery basket, the ascot row and its reference swap leg, the whole option exercise date list, the callable bond's call dates as child rows, the convertible bond's conversion ratios as child rows, and the ten-code coverage table.

xml_bond_source_fidelity_tests is new and answers a different question. The golden suite compares our output to a stored copy of our own output, so it catches drift and nothing else. This suite reads the ORE documents themselves, canonicalises each one, reads the canonical form and canonicalises it again, and requires the two canonical forms to be identical. Eleven source documents state a bond trade and all eleven are stable. The suite reads the whole examples directory to find them, so the count is 132 documents scanned.

The golden suite gains Credit_CallableBond.xml, the ninth bond product with a stored golden.

Wave 2.3 added the BondFuture and Ascot arms to the importer and left the exporter's reverse dispatch at seven bond types. A future or an ascot therefore counted in trades_mapped and never reached the output document, because the dispatch logged "No reverse mapper for bond type" and returned. No test covered it: the two products have no suite-level roundtrip case.

Two exporter cases now cover them. export_portfolio_ascot_roundtrip drives Cash_Ascot.xml through the importer and the exporter and requires AscotData in the output. export_portfolio_bond_future_roundtrip builds the trade the tree cannot supply, maps it, exports it and requires BondFutureData.

The acceptance asks for the written document to differ from the source only in ordering and representation. scripts/ore_mapper_roundtrip_diff.py is the check. It compares our output to the source document as a multiset of (element path, value) pairs, so ordering is invisible by construction, and it classifies every remaining pair.

The comparison below runs the stored goldens against the sources they came from, which is the domain path (save_data(load_data(document))). The mapper path is a different pair of functions, and its diff belongs to wave 2.5, which runs exporter::roundtrip over the tree. What follows is the domain result, and it sets the baseline the mapper path has to meet.

Over the 132 documents of the examples tree:

• 543 pairs differ in numeric spelling only, with a worst relative error of 5.15e-8. The ORE schema types these fields as xs:float and the example documents state more digits than that type holds, so the source over-specifies its own schema and our output is the correctly typed value.
• 25 pairs differ in boolean spelling only. The schema's bool type enumerates Y, YES, TRUE, True, true and 1; the examples spell true as an empty element, which is not one of them. Our output writes Y.
• No pair is lost. No pair is unexplained.

The first run of the new suites failed two cases, both on the fixtures the suite authors rather than on the mapper. CallData requires IncludeAccruals, and totalReturnData requires ScheduleData. Both documents now carry them. The failures are worth recording because they show the generated parser enforces the schema's minimum occurrence rules, which is the property the fixtures rely on.