Task: Sync ORE examples and XSD to v1.8.17.0
Table of Contents
This page documents a task in the Update ORE to v17 story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Sync the vendored examples and XSD from the v1.8.17.0 engine checkout,
and regenerate the C++ domain types from the new XSD. The known
upstream hand-patches must survive the sync. Report whether
TA002_IR_Swap.tar.gz changed.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Update ORE to v17 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-09-12 |
3. Acceptance
external/ore/examples/andexternal/ore/xsd/match the v1.8.17.0 engine checkout, with provenance recorded in theirmanifest.json.- The known upstream hand-patches are reapplied.
- The C++ domain types regenerate from the new XSD with no unexplained diff.
TA002_IR_Swap.tar.gzis byte-identical, or the change is explained and accepted.
4. Plan
The sync follows the scripted methodology in external/ore/methodology.txt.
The plan covers the two points where that methodology does not apply as
written.
- Sync the examples and XSD with
external/ore/tools/update_ore_samples.sh, which also regenerates bothmanifest.jsonfiles and reapplies the known hand-patches. - Regenerate the domain types with
projects/ores.codegen/scripts/xsdcpp_generate_ore.sh, then runclang-format -itwice over the two generated C++ files. The committed files are formatted and raw xsdcpp output is not, so one pass leaves a large phantom diff. The second pass is a fixed point and proves it. - Verify on a committed tree: the two codegen drift checks, the ores.ore test binary, and the example coverage test added here.
5. Notes
- The
v1.8.17.0tag on the engine fork is provisional. Upstream had not tagged when this work started, so the tag was created at the sync commit,3d75a6908. The package rebuilds once upstream tags properly. TA002_IR_Swap.tar.gzis byte-identical to its previous contents. Checked by extracting it and diffing itsInput/tree againstexternal/ore/examples/Academy/TA002_IR_Swap/Input/.- The sync deletes 13 files whose names look like
#run.py#. They are Emacs auto-save files that a previous hand-vendoring committed by accident. The engine checkout has none, and the sync mirrors that checkout, so they go. They are the only deletions in the diff. - The
oreTradeDatablocker recorded indoc/agile/product_backlog/next/xsdcpp-fails-to-resolve-oretradedata-group.orgdid not recur. The script that failed during the v16 bump ran clean. - Seven trade types are new in v1.8.17.0, and none of them has a mapper
or a round-trip test:
BondFutureOption,CommodityForwardVolatilityAgreement,EquityAutoDeltaHedgedOption,EquityForwardVolatilityAgreement,ForwardVolatilityAgreement,FxForwardVolatilityAgreement, andIntradayPowerForward. OnlyIntradayPowerForwardhas a portfolio example, atexamples/MarketRisk/Input/Intradaypower/portfolio.xml. Tracked by Cover the seven new ORE v1.8.17.0 trade types. Credit_CallableBond.xmlwas the one example trade with no golden and no test reference. It predates v17. The coverage test added here fails on any such gap from now on, and this file's golden is committed with it.assets/test_data/domain_roundtrip/is the second derived artefact of the examples tree, and it is now out of step with it. A fresh run ofores.cli ore roundtripagainst the new examples tree writes 566 files; the committed dataset holds 1474. Regenerating would change about 1190 paths: 961 deletions, 53 additions and 177 changes. The deletions are 946 emptyPortfoliostubs that an older build wrote for every XML it could not parse, plus 15 orphans forProducts/SupportedTrades/, a directory upstream removed. The additions are the examples this sync introduced. The changes are mostlyconventions.xmlandcurveconfig.xml.- That regeneration is deliberately not part of this task. It would
dominate the diff and bury the sync. It also cannot be decided on its
own, because
golden_datasetduplicates this dataset and sprint 17 claimed to have deleted it. Tracked by Regenerate or retire the domain round-trip dataset. scripts/ore_domain_roundtrip_check.pyis the only consumer, and CI runs it without--strictwhileassets/**sits in that workflow'spaths-ignore. So the staleness fails no build and triggers no run.- The XSD regeneration broke the build in three places.
commodityOptionDatareplaced its inlineStrikeelement with thestrikeGroupsubstitution group, socommodity_instrument_mapperstopped compiling ond.Strike. Fixed by reading the strike fromd.strikeGroupand writing it back the way the bond and equity mappers already do. AStrikeDatastrike is read as well, because the new schema allows it and reading onlyStrikewould drop it silently. - The second break is in
fx_instrument_mapper, and it is larger. Five FX types changedOptionDataandBarrierDatafrom a repeated element to a single one:fxBarrierOptionData,fxDigitalOptionData,fxDigitalBarrierOptionData,fxDoubleBarrierOptionData(through the shared barrier type), andfxTouchOptionData. Everypush_backon those members stopped compiling, in both directions, so the mapper carries one read helper and one write helper for each. A double barrier now keeps both levels in the one group, lower first, which is how upstream writes them. - The third break is small and it hid behind an unchanged member name.
sensitivityanalysis.DiscountCurvesbecamexsd::optional<discountcurves>, but it kept the name, so the name diff did not flag it.xml_sensitivityanalysis_roundtrip_testsread through it with.DiscountCurveand stopped compiling. Fixed by copying the optional guard that theIndexCurvescheck on the next line already used. fxKIKOBarrierOptionDataandgenericBarrierOptionDatakeep their nestedBarriers/BarrierDatavector and were not touched.- Both breaks were found by the compile-database syntax check described below, which is what made them cheap to fix while the build lock was held by another worktree.
- Upstream restructured several other types. No ORES code reads them, so
nothing else broke:
bondFutureDatalost sixteen members,stFreeStyleEventScheduleBasereplaced its members withScheduleDataandDerivedSchedule, andIborIndexCurvebecameIndexCurveon the bond curve config types. - A member-name diff found the first two breakages, which is worth
repeating on the next bump. Diff every complexType's member names
between the pre-regeneration commit and the regenerated
domain.hpp, then grep for each removed name. That finds a removed member before a build does. - That diff misses the third break, because the member name did not
change.
sensitivityanalysis.DiscountCurveswent from a requireddiscountcurvestoxsd::optional<discountcurves>, andxml_sensitivityanalysis_roundtrip_testskept calling.DiscountCurvethrough the optional. Diff the member types as well as the names, not only the names. Comparing types on the 844 complexTypes found 31 members that changed type or vanished, of which 13 are named by hand-writtenores.orecode. - Cardinality is the break class to watch. Both the FX change (repeated to single) and this one (required to optional) left the member name alone and moved only the type.
- A second lever checks the fixes without a build. Read the command for
one source file out of
build/output/linux-clang-debug-make/compile_commands.json, remove-cand-o, add-fsyntax-only, and run it. That type-checks one translation unit in seconds and needs no build-lock slot. A sweep over the 32ores.oresource units and the 59 test units found the second break class before the queued build reached it. Reuse it on the next bump: it is the only local check that does not queue behind the lock.
6. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario, run
through the QA Validation Runner panel) that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
7. PRs
| PR | Title |
|---|---|
| #2072 | [ore] Sync ORE examples and XSD to v1.8.17.0 |
8. Review
Local code review against the Code Review Checklist, run before the PR was raised. No critical findings.
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | make_barrier_group treats lower = 0.0= as "no lower barrier", so a barrier level of exactly zero would be dropped |
fx_instrument_mapper.cpp |
Declined | The convention predates this change: the code it replaces also compared instr.lower_barrier ! 0.0=. An FX barrier at exactly zero is not a meaningful level, and the helper now documents the rule. |
| 2 | The new coverage test is a thirteenth consumer of assets/test_data/golden_dataset/, a dataset whose retirement the sprint_17 task claims to have completed |
xml_example_coverage_tests.cpp |
Accepted, deferred | Real, and wider than this test. Recorded in Regenerate or retire the domain round-trip dataset. |
| 3 | The twelve golden suites print "run with --reset-goldens to bootstrap", but that flag only deletes the dataset; nothing writes it back |
xml_*_golden_roundtrip_tests.cpp |
Accepted, deferred | Pre-existing, and not this PR's to fix. Recorded in –reset-goldens deletes the golden dataset and nothing recreates it. |
| 4 | The coverage test walks Products/Example_Trades only, and non-recursively |
xml_example_coverage_tests.cpp |
Declined | That directory holds the trade examples. The twelve suites cover the rest of the examples tree, and 132 files is the whole trade set. |
| 5 | std::to_string writes a strike as "100.000000", not "100" |
commodity_instrument_mapper.cpp |
Declined | Byte-for-byte the idiom the bond and equity mappers already use, and the round-trip compares parsed values, so trailing zeros do not reach a test. |
| 6 | Misspell check (misspell-fixer .) exits 1 on a clean tree |
whole tree | Fixed | Every hit sat in the untracked, gitignored projects/ores.codegen/venv. The ignore file names the compass venv but not this one. Added the missing line in b772be2e43; a run from the repo root now reports "nothing to replace" and exits 0. |
| 7 | LOAD_PROFILE_1_EXPLICT is a misspelling |
examples/MarketRisk/Input/Intradaypower/{conventions,portfolio}.xml |
Declined | An upstream data identifier, not prose. It names a load profile and the portfolio references it by that name, so correcting the spelling breaks the example. Present verbatim in the engine checkout. |
9. Result
DONE. The examples and XSD trees match the v1.8.17.0 engine checkout, the
C++ domain types regenerate from the new XSD, and the three compile
breaks the regeneration caused are fixed. TA002_IR_Swap.tar.gz is
byte-identical to its previous contents.
The branch mixes code, docs and ci, so the union of the three check sets
ran against b772be2e43:
| Check | Result |
|---|---|
Build, linux-clang-debug-make |
clean |
ctest |
71/71 passed, 0 failed, 777.59 s |
Round-trip, scripts/ore_domain_roundtrip_check.py |
exit 0 |
| cmake-sources drift | clean, exit 0 |
| Codegen component drift | "No drift: regenerated output matches the checked-in tree", exit 0 |
| Site build | "Build succeeded.", 5318 nodes / 31901 links |
Misspell, misspell-fixer . |
exit 0, "nothing to replace" |
The first ctest round failed eight suites, and none of it was code.
Three of them named a column or a row the schema had dropped, because
the worktree database was 15 days behind HEAD. compass db status
reported that drift and named its own remedy. The recreate fixed those
three. The other five failed on NATS connect failed, because compass
services stop preceded the recreate and nothing restarted the broker.
With the 23 services back up, all 71 suites pass.
The coverage test added here walks all 132 files in
Products/Example_Trades and finds a golden and a test reference for
each one. Credit_CallableBond.xml was the single gap; its golden is
committed with it.