Task: Refresh ORE Engine binary package, examples, and XSD to 1.8.16.0
Table of Contents
This page documents a task in the Update vendored ORE Engine to 1.8.16.0 story. It captures the goal, current status, acceptance, and any notes or results.
Goal
Fix the broken (not self-contained) vendored ore-1.8.15.0 package and refresh external/ore/examples and external/ore/xsd to a current ORE Engine build, with tooling so both refreshes are reproducible and safe to repeat.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Update vendored ORE Engine to 1.8.16.0 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-07-30 |
Acceptance
- external/ore/packages/ore-VERSION-x64-linux.tar.gz is genuinely self-contained (ldd resolves every dependency inside the package, none pointing at a build tree) and passes an automated self-test (runs TA002_IR_Swap, tolerantly diffs Output against ExpectedOutput, writes a sign-off diff-report.txt); external/ore/examples and external/ore/xsd are synced from the same engine commit and pass scripts/validate_ore_examples.sh; the 3 known upstream FX-key hand-patches (PR #1423) are reapplied automatically by the sync script, not manually; all new/changed tooling is committed as reusable scripts, not one-off manual steps.
Plan
Cloned/synced the ORE Engine fork (git@github.com:OreStudio/Engine.git,
upstream OpenSourceRisk/Engine) to ~/Development/ORE/Engine,
built it (linux-clang-ninja-release, system Boost 1.90.0), then
worked in three parts:
- Packaging (
external/ore/tools/package_ore.sh): the previousore-1.8.15.0-x64-linux.tar.gzshipped the executable alone, still dynamically linked againstlibOREAnalytics.so=/=libQuantExt.so=/ =libQuantLib.so.1with no RPATH into the package –lddreported them "not found", andores.compute.wrapperspawns the packaged executable directly with noLD_LIBRARY_PATHcustomisation, so it could only ever have worked by coincidence. New script bundles every non-glibc runtime dependency underlib/, rewrites RPATH to$ORIGIN(/lib)viapatchelf, and self-tests by actually running the package againstTA002_IR_Swapand tolerantly diffing (seeexternal/ore/tools/compare_csv.py– cross-compiler float noise and legitimate schema evolution, e.g. new report columns, are not failures) against the realExpectedOutput/, writing a sign-offdiff-report.txt. - Examples + XSD sync (
external/ore/tools/update_ore_samples.sh): syncsexternal/ore/examples/andexternal/ore/xsd/together from the same engine commit (a version mismatch between them produces spurious schema-validation failures unrelated to either side being wrong – found this the hard way when the oldxsd/didn't recognise the newer examples'ExtrapolationMethodelement), regenerates amanifest.jsonin each with a reproducibletree_sha256, and runsscripts/validate_ore_examples.shautomatically. Fixed a real, pre-existing bug in that validate script along the way (=set -e=ing out on the first genuine validation failure before its own pass/fail accounting ever ran). - Hand-patch preservation: caught, before committing, that the
fresh rsync silently reintroduced a genuine upstream bug (3 files'
FX/RATE spot-quote keys ship in the wrong order per QuantLib's own
documented
FXSpotQuoteconvention – confirmed independently, not just trusted from history) that this codebase had already fixed once in PR #1423 but never automated reapplying. Addedreapply_known_patches()to the sync script so this can't regress silently again; verified idempotent across three re-syncs. Also found and fixed a second sync-script bug the same way:rsync --deletewas deleting this directory's own hand-writtenREADME.md=/=methodology.txt(they don't exist in the upstream source, so a plain sync sees them as "removed upstream").
Deliberately did NOT run the XSD-driven C++ codegen step
(xsdcpp_generate_ore.sh) to regenerate
projects/ores.ore/include/ores.ore/domain/domain.hpp=/=domain.cpp –
the local xsdcpp build fails (error: Could not find group
'oreTradeData') against both the old and new schema (confirmed via a
clean A/B test), a pre-existing environment/tool-version issue
unrelated to this task's own changes. Filed as a capture rather than
worked around blind; domain.hpp=/=domain.cpp are left unregenerated,
a known, explicitly-flagged gap.
Notes
- Package version
1.8.16.0-3-g3b62ba248isgit describe --tags --alwaysagainst the Engine checkout (3 commits past thev1.8.16.0tag) – seeexternal/ore/packages/manifest.jsonfor the full provenance/version-scheme explanation. - Filed captures for out-of-scope findings along the way: the xsdcpp
blocker (
99A443B8), a compute-grid package integrity gap (no SHA256 verification on download,8DBD1BAE), and ORE's PythonTradeGeneratoras a possible building block for Acme's synthetic trade portfolios (2521B2BD).
Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario, run
through the QA Validation Runner panel) that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
PRs
| PR | Title |
|---|---|
| #1786 | [data] Update vendored ORE Engine to 1.8.16.0 |
Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | compute_ore_app_seed.sql still seeds the deleted ore-1.8.15.0 package (engine_version, package_uri) | compute_ore_app_seed.sql | Fixed | Updated all 5 references to 1.8.16.0-3-g3b62ba248; verified via db recreate that the new engine_version/package_uri land correctly. |
| 2 | reapply_known_patches() doesn't verify its sed actually matched | update_ore_samples.sh | Declined (informational) | Real robustness suggestion for a future upstream format change, not a defect in this PR's actual output (verified patches applied correctly); left as a known, minor follow-up. |
| 3 | 57.5MB binary committed as a plain git blob, no LFS | ore-1.8.16.0-3-g3b62ba248-x64-linux.tar.gz | Accepted, no change | Matches this repo's existing convention – no Git LFS anywhere, other external/* dirs already vendor large binaries (mddl PDF, lei xlsx, etc.). |
| 4 | compare_csv.py compares rows positionally, not by key | compare_csv.py | Declined (informational) | ORE's own CSV row order is deterministic for a given input; noted as a future-proofing concern only, not a current defect. |
Result
Replaced the broken ore-1.8.15.0-x64-linux.tar.gz with a genuinely
self-contained, self-tested ore-1.8.16.0-3-g3b62ba248-x64-linux.tar.gz
(verified: ldd resolves every dependency inside the package with
none pointing at a build tree; automated run against TA002_IR_Swap
tolerantly matches real ExpectedOutput/ within rtol=1e-6, with
schema differences called out explicitly for sign-off). Synced
external/ore/examples/ (3404 files) and external/ore/xsd/ (26
files) from the same engine commit, both with a fresh manifest.json
recording exact provenance; all 1566 example XML files validate
against the refreshed schema. Automated the 3-file FX-key hand-patch
reapplication that a fresh sync had previously silently undone (caught
live, fixed, verified idempotent) – resolved the long-deferred capture
this had been tracked under. Added two new reusable scripts
(external/ore/tools/package_ore.sh, update_ore_samples.sh, plus
compare_csv.py) and their own README/manifest/methodology docs,
matching this repo's existing external/<source>/ vendoring
convention. Fixed a genuine pre-existing bug in
scripts/validate_ore_examples.sh (set -e exiting before its own
pass/fail accounting on the first real validation failure) found while
using it. C++ domain-type codegen (xsdcpp_generate_ore.sh) could not
be run – local xsdcpp build fails against both old and new schema,
a pre-existing environment issue unrelated to this task, filed
separately rather than worked around.