Sprint 22
Table of Contents
This page documents a sprint (Sprint 22) of ORE Studio. It captures the sprint's mission, current status, and the stories that compose it. For the surrounding context — version goals, sprint order, and product identity — see Version 0.
Mission
Continue commissioning the ores.refdata entity library and correcting codegen C++ generation drift, carrying postponed sprint 21 stories.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent version | Version 0 |
| Previous | Sprint 21 |
| Start | 2026-07-02 |
| End | 2026-07-10 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Open sprint 23. |
| Release Notes | release_notes.org |
| Last touched | 2026-07-10 |
Achievements
- Shipped the Currency Pairs & Conventions reference-data entity end to end: knowledge-base foundation, domain/db/codegen model, Qt UI (list, detail, duplicate/inverted-pair rejection, Conventions cross-navigation), and a full user-manual chapter with captured screenshots.
- Proved out the org-mode literate codegen model on
party(PR #963) — a single org file now drives both C++ and SQL generation with inline custom repository methods, replacing the two-JSON-file pattern; the mechanism is now the template for future entity migrations. - Landed
ores.orgmode, a new C++ component parsing org-mode docs and resolvingid:links via the existing org-roam index, and built the QA Validation Runner on top of it — an in-app Qt panel turning atest_scenariodoc into a guided, trackable test session across the multi-worktree fleet. - Added codegen support for view groups (Qt detail-dialog tabs), piloted on currency, replacing the single-tab-only generated dialog layout for entities with many fields.
- Closed out a cluster of Windows/macOS CI reliability issues: the MSVC C4273 dllimport failure (via a new ores.qt.headless extraction), a stale-Qt-plugin-after-ABI-change crash, missing EXPORT macros across 20 refdata/scheduler service classes, and the macOS relevance-guard silently skipping the build matrix.
- Fixed a real multi-tenant correctness bug: tenant-scoped
variability system settings other than
bootstrap_modeweren't seeded per-tenant, silently hiding gated features for every non-system tenant. - Found and fixed the NATS-port-inside-the-kernel-ephemeral-range bug
(moved
BASE_PORT_STARTbelow 32768) after it caused a real bind collision during this sprint's own manual-chapter work — .env format version bumped to 11. - Ran three System 2 health reviews (day 3, day 4, close) tracking the sprint honestly through two consecutive RED verdicts to a third; the close review's findings (8 stale sprint.org rows, 20 simultaneously STARTED stories, 741 commits against a 300 target) directly drove this closure's scope: 43 of 70 stories carried back to the product backlog rather than force-closed or silently dropped.
Stories
For the definitions of the themes see Themes.
Agile
| Story | State | Start | End | Description |
|---|---|---|---|---|
| Open sprint 22 | DONE | 2026-07-02 | 2026-07-10 | Scaffold sprint 22, wire manifest, move postponed stories. |
| Fix Windows dllimport build failure via ores.qt.headless extraction | DONE | 2026-07-08 | 2026-07-09 | MSVC C4273 broke Windows CI; root cause was Widgets-free logic living inside ores.qt.api, fixed by extracting a proper ores.qt.headless component. |
| Investigate RegulatoryBookTypeController Windows LNK2019 error | DONE | 2026-07-09 | 2026-07-09 | Root-caused a pasted Windows LNK2019; no defect found on current main, no speculative fix made. |
Product
Epic: ores.refdata Commissioning
Commission all ores.refdata entities across every access layer: verify Qt UI end-to-end
post-NATS migration (list window, detail window, history, delete, eventing), implement or
verify shell and CLI commands (list, add, remove), document each entity in the user manual,
and file backlog captures for Wt and HTTP gaps. One story per entity. Carried from sprint 21.
| Story | State | Start | End | Description |
|---|---|---|---|---|
| Commission: currency | DONE | 2026-07-10 | 23/27 tasks done; remaining 4 abandoned (superseded by top-level shell/CLI/Wt/HTTP stories) or found already complete (badge rendering, stale bookkeeping). | |
| Reconcile currency Qt custom features with codegen | DONE | 2026-07-04 | 2026-07-04 | Triaged currency's custom Qt functionality vs. the qt codegen profile, piloted the combo-field mechanism, and produced the regenerate plan for the Qt sync task. |
| Commission: rounding_type | DONE | 2026-07-10 | Aux type: verify SQL, sync all codegen profiles, verify Qt UI, document in currency chapter. | |
| Commission: monetary_nature | DONE | 2026-07-10 | Aux type: verify SQL, sync all codegen profiles, verify Qt UI, document in currency chapter. | |
| Commission: currency_market_tier | DONE | 2026-07-10 | Aux type: verify SQL, sync all codegen profiles, verify Qt UI, document in currency chapter. | |
| Commission: party | ABANDONED | 2026-07-02 | 2026-07-10 | Verify Qt; implement shell + CLI; manual; Wt/HTTP captures. |
| Commission: party_type | DONE | 2026-07-04 | Verify Qt; implement shell + CLI; manual; Wt/HTTP captures. | |
| Commission: party_status | ABANDONED | 2026-07-06 | 2026-07-10 | Verify Qt; implement shell + CLI; manual; Wt/HTTP captures. |
| Commission: party_id_scheme | ABANDONED | 2026-07-10 | Verify Qt; implement shell + CLI; manual; Wt/HTTP captures. | |
| Commission: contact_type | ABANDONED | 2026-07-10 | Verify Qt; implement shell + CLI; manual; Wt/HTTP captures. | |
| Commission: counterparty | ABANDONED | 2026-07-10 | 3/3 codegen-hardening tasks done; Qt/shell/CLI/manual scope still open. | |
| Commission: book | DONE | 2026-07-02 | 2026-07-10 | Verify Qt; implement shell + CLI; manual; Wt/HTTP captures. |
| Commission: book_status | DONE | 2026-07-06 | 2026-07-10 | Verify Qt; implement shell + CLI; manual; Wt/HTTP captures. |
| Commission: business_unit | ABANDONED | 2026-07-10 | Verify Qt; implement shell + CLI; manual; Wt/HTTP captures. | |
| Commission: business_centre | ABANDONED | 2026-07-10 | Verify Qt; implement shell + CLI; manual; Wt/HTTP captures. | |
| Commission: portfolio | ABANDONED | 2026-07-10 | Verify Qt; implement shell + CLI; manual; Wt/HTTP captures. | |
| Commission: purpose_type | DONE | 2026-07-02 | 2026-07-10 | Verify Qt; implement shell + CLI; manual; Wt/HTTP captures. |
| Commission: business_day_convention_type | DONE | 2026-07-04 | 2026-07-07 | Move from ores.trading to ores.refdata (zero current consumers), then commission fully to Qt; spun off from the currency pair story. |
| Commission: day_count_fraction_type | ABANDONED | 2026-07-10 | Move from ores.trading to ores.refdata (update trading_swap_legs' one reference), then commission fully to Qt; spun off from the currency pair story. | |
| Move book, book_status, and portfolio to ores.trading | ABANDONED | 2026-07-10 | These are trading/booking structures, not static reference data; relocate full stack out of ores.refdata. | |
| Codegen: generate per-entity NATS event-mapping registrar | DONE | 2026-07-08 | Hand-wired application.cpp eventing registration caused a missed-wiring regression (purpose_type); generate it per-entity instead. | |
| Regen entities with latest codegen templates | DONE | 2026-07-09 | 2026-07-10 | Country regenerated and reconciled against the shared codegen templates' new capability landed via currency's commissioning; remaining entities recaptured for future pickup. |
| Book data model cleanup | ABANDONED | 2026-07-08 | 2026-07-10 | 2/9 tasks done (the 2 live bugs); add classification flags, rename ledger_ccy, investigate legal-entity/branch fields, add rates centre, analyse allowed lists, enrich manual with domain concepts remain. |
| Rethink synthetic reference-data generation across entities | ABANDONED | 2026-07-10 | Currency's ad-hoc Generate feature (client-side blue-highlighted unsaved rows, per-entity hand-wiring) is half-baked and not actually save-able; design a proper cross-entity mechanism for single + batch synthetic generation before reintroducing it anywhere. |
Epic: Market data generation
Build out synthetic market data generation infrastructure across the stack — SQL variability, entity models, and librarian-driven per-party configuration — starting from FX and generalising to other asset classes over time.
| Story | State | Start | End | Description |
|---|---|---|---|---|
| Codegen SQL variability for hypertable, bi-temporal and GIST patterns; model ores.marketdata entities | DONE | SQL template variability (hypertable, soft-update/soft-delete triggers, GIST exclusion) + .org models for market_series, market_observation, market_fixing; prerequisite for full-stack codegen of ores.marketdata. | ||
| Market data cleanup: retire dead duplicate tables and preserve hand-written overrides | ABANDONED | 2026-07-10 | Drop legacy pre-codegen ores_marketdata_series/observations/fixings_tbl (superseded, zero C++ references); remove their stale validation_ignore.txt entries; preserve market_observation's hand-written repository/service overrides against codegen regeneration. | |
| Temporal composite entity versioning | DONE | 2026-07-06 | 2026-07-07 | Design doc: child saves (e.g. party_identifiers) must bump the parent's version; loading a parent as-of version N must compose children as of that version. Work out enforcement across SQL/codegen and Qt composite version/history UX. Found via party LEI/BIC identifiers not bumping party.version. |
| Implement temporal composite entity versioning | DONE | 2026-07-07 | 2026-07-10 | Implement the target state: SQL touch-function mechanism, as-of window-join read-side composition, Qt composite history dialog. Party/counterparty are the reference case. |
| Full market.txt import hits duplicate observation key | DONE | 2026-07-04 | 2026-07-04 | Fixed: the file genuinely repeats ~100 (date, key) pairs, not a series_key_registry bug; import_service now de-duplicates last-line-wins with a warn/error policy; unblocks the 2016-02-05 vintage import. |
| Configurable feed status indicator in the FX spot grid | DONE | 2026-07-02 | 2026-07-03 | FX spot grid status pips use hardcoded CSS colours; replace with a configurable indicator once the right UX (badge vs. compact status dot) is settled. |
| GMM improvements: tidy up synthetic data generation loose ends | DONE | 2026-07-02 | 2026-07-03 | Chart axis labels, per-component + combined PDF display, and stochastic process component gaps (incl. Ornstein-Uhlenbeck). |
| IR Rates synthetic data generation | ABANDONED | 2026-07-10 | Short-rate stochastic process + Curve Template raw instrument generation (deposits/FRAs/swaps), tenor-collision validation, tick-batch publishing — no bootstrapping, no vol surfaces, no QuantLib. | |
| Synthetic data librarian support: FX foundation | DONE | 2026-07-02 | 2026-07-03 | Librarian gives each new party a usable synthetic data configuration; FX only for now, but db tables/publishing built generically via codegen for future asset classes. |
| Synthetic data collections: Basic and Realistic | ABANDONED | 2026-07-03 | 2026-07-10 | Two sample FX collections seeded from real ORE reference data (2016-02-05 vintage), Market Simulator grouping by collection, feed namespacing, vintage-availability guard. |
| ores.shell command for market data import | DONE | 2026-07-04 | 2026-07-04 | Non-interactive market data import command, reusing the existing import_service; unblocks the synthetic data collections story's vintage import task. |
| Move trade import orchestration to service layer | ABANDONED | 2026-07-10 | Move ImportTradeDialog's ~300 lines of trade-import orchestration (XML parse, per-trade save, 30-way instrument dispatch) into a server-side trade_import_service + import_trades_request, so ores.shell's future trade import command doesn't duplicate it. | |
| Currency pair support in reference data | DONE | 2026-07-06 | 2026-07-10 | Knowledge-base foundation (conventions, quoting, cross rates) followed by domain/db/codegen support for currency pairs as reference data. |
Tooling
Epic: Codegen
Codegen drift correction and guardrails: resolve template-production divergence, land safety and CI checks so generated code stays trustworthy. Carried from sprint 21.
| Story | State | Start | End | Description |
|---|---|---|---|---|
| Decommission ores.codegen.table model type | ABANDONED | 2026-07-10 | Migrate all remaining _table.org models to ores.codegen.entity; remove the table type from the facet catalogue. | |
| Refactor ores.codegen C++ generation | ABANDONED | 2026-07-06 | 2026-07-10 | Audit/resolve template-production drift; carried from sprint 21. |
| Decommission legacy codegen bash scripts | ABANDONED | 2026-07-10 | Replace direct bash invocations with compass or Python codegen; remove the scripts. | |
| Codegen CI zero-diff invariant | ABANDONED | 2026-07-10 | CI job regenerating all registered components, failing on any diff vs HEAD. | |
| Codegen model safety guardrails | ABANDONED | 2026-07-10 | Runtime guards for missing component_include/component_core; block dual-template SQL generation. | |
| Codegen unified model — Phase 1: Qt derivation | ABANDONED | 2026-07-10 | Derive Qt fields from conventions; drop explicit NATS protocol class names from models. | |
| Codegen unified model — Phase 3: unify temporal templates | ABANDONED | 2026-07-10 | Merge temporal and non-temporal C++ template families into single model-controlled templates. | |
| Codegen developer experience improvements | ABANDONED | 2026-07-10 | Fix spurious ERRORs in –component mode; add –entity filter; add entity group / tag system for domain-scoped generation. | |
| Introduce per-entity NATS sub-registrars | DONE | 2026-07-04 | Replace monolithic registrar.cpp with generated per-entity sub-registrars; isolate handler compilation; unblock incremental service API migration. | |
| Resolve codegen model unification blockers | ABANDONED | 2026-07-06 | 2026-07-10 | Carried from sprint 21. |
| Composite child-entity and hierarchy Qt widgets for codegen | DONE | 2026-07-04 | 2026-07-10 | Handcrafted HierarchyTreeWidget + child-entity table facet, paste-block-bound, so party/counterparty can retire their hand-maintained EntityDetailDialog. Overlaps with solid_dirac's Commission: party work — compare approaches before implementing. |
| Codegen support for view groups (Qt detail-dialog tabs) | DONE | 2026-07-09 | 2026-07-10 | The pre-codegen currency dialog grouped fields into separate tabs; codegen's Qt detail-dialog template only ever generates a single tab, making entities with many fields harder to read. Add a generic view_group concept to the entity_org schema and Qt codegen templates, piloted on currency, and document the domain knowledge in the manual. |
Epic: Compass
| Story | State | Start | End | Description |
|---|---|---|---|---|
| compass generate catalogue | ABANDONED | 2026-07-10 | Auto-regenerate catalogue/index org files by scanning tagged org files. | |
| Fix orphan documents and tag taxonomy | DONE | 2026-07-06 | 2026-07-10 | Fix orphaned docs in org-roam graph; create tag inventory; add compass tag validation. |
| Compass quality of life (Sprint 21) | DONE | 2026-07-02 | Carried from sprint 21. | |
| Add compass client start/stop subcommands | DONE | 2026-07-03 | 2026-07-03 | compass client was flat (launch-only); add start/stop matching the services/env/nats subcommand convention. |
| Compass GitHub Actions support | ABANDONED | 2026-07-10 | Give compass a quick overview of all GitHub Actions runs, highlight which ones are failing, and for failing runs surface the error detail via gh CLI without needing to download full logs. |
Epic: Org-mode support
Add first-class org-mode support to the C++ side of the codebase — parsing docs into a structure, resolving id: links — as a shared foundation for tooling built on top of it. First two stories: ores.orgmode itself, then a Qt QA Validation Runner built on it to address the fleet's testing bottleneck (10 worktrees, one human tester, no way to track what's under test or what was found). More stories expected here over time as other consumers emerge.
| Story | State | Start | End | Description |
|---|---|---|---|---|
| ores.orgmode: C++ org-mode parsing and org-roam link resolution | DONE | 2026-07-06 | 2026-07-07 | New component: parse org-mode docs into a C++ structure, resolve id: links via the existing org-roam SQLite index. Pure C++, usable from ores.shell; ports the shape of ores.compass's Python org parser. |
| QA Validation Runner: in-app test tracking for the multi-worktree fleet | DONE | 2026-07-06 | 2026-07-10 | Qt panel, built on ores.orgmode, turning a test_scenario org doc's checklist into a guided test session; results and story/task context rendered in-app, not a standalone JSON file. |
Infrastructure
| Story | State | Start | End | Description |
|---|---|---|---|---|
| Verify Windows and macOS CI builds | ABANDONED | 2026-07-10 | Re-run MSVC and macOS builds; verify the rfl C1202 fix; fix or file follow-ups. | |
| Upgrade Qt to latest supported version to fix macOS 15 AGL removal | ABANDONED | 2026-07-06 | 2026-07-10 | Qt 6.8.x references -framework AGL (removed in macOS 15); the latest supported Qt drops that dependency. CI pinned to macos-14 as stopgap. |
| Canary CI: near-zero sccache hit rate makes every build a cold rebuild | DONE | 2026-07-03 | 2026-07-04 | Cache-quota fix applied but doesn't explain the magnitude (32 files changed vs. 98.6% miss rate); running a live zero-source-diff experiment. |
| Fix NATS port collision with legacy env and label inconsistency | DONE | 2026-07-03 | 2026-07-03 | solid_dirac's NATS ports collided with a stray legacy env's server; migrate to base-port-derived NATS ports and unify the config label to underscore form. |
| Fix uninitialized asset_class/series_subclass in market_series generator | DONE | 2026-07-03 | 2026-07-03 | Nightly Valgrind flagged uninitialized-memory defects in market_series_mapper::map(); generate_synthetic_market_series() never set the two enum fields. |
| Fix tenant-scoped variability system settings: permissions and per-tenant seeding | DONE | 2026-07-10 | IAM can't clear a tenant's bootstrap_mode (missing DB grant, silent permission-denied); tenant-scoped system settings other than bootstrap_mode aren't seeded per-tenant, so gated features silently vanish for non-system tenants. |
Documentation
| Story | State | Start | End | Description |
|---|---|---|---|---|
| Improve site documentation discoverability | ABANDONED | 2026-07-10 | Carried from sprint 21. |
Hotfixes
| Story | State | Start | End | Description |
|---|---|---|---|---|
| Hotfix: MacOS CI relevance-guard skips matrix build silently | DONE | 2026-07-04 | 2026-07-04 | Continuous MacOS workflow reports overall Success while the scheduled relevance-guard skips the build matrix, so macOS builds are not actually being verified. |
| Hotfix: refdata generator/repository test fixtures out of sync with DB constraints | DONE | 2026-07-04 | 2026-07-05 | Root cause: 2 counterparty codegen-facet commits introduced generator values violating DB constraints and test expectations; fixed and migrated touched literals to shared change-reason constants. |
| Hotfix: stale Qt plugin causes crash after ores.qt.api ABI change | DONE | 2026-07-08 | 2026-07-08 | CMake had no dependency edge from ores.qt.exe to the QPluginLoader-loaded plugin libs, so an ABI change could leave a stale plugin .so segfaulting on load. Cherry-picked the fix off feature/vintage-guard onto its own branch. |
| Fix missing ORES_REFDATA_CORE_EXPORT on refdata/scheduler service classes | DONE | 2026-07-09 | 2026-07-09 | 20 service classes across ores.refdata.core and ores.scheduler.core were missing their EXPORT macro, so with -fvisibility=hidden their methods were invisible outside their own .so — causing undefined-reference linker errors for any cross-library caller (e.g. ores.iam.core.tests calling party_service/counterparty_service). |
Achievements
Milestones worth capturing as they happen, so the release notes at
sprint close have the evidence to hand. Add a dated ** subheading
per milestone, with any screenshot via [[proj:assets/images/<file>.png]].
Captures
Real captures filed during this sprint that have not been pulled into a
story as tasks — listed here purely for visibility into what the sprint
surfaced. Each is an inbox capture (compass capture); triage them at the
health review (promote to a story, move to next, or discard).
| Capture | Tags | Description |
|---|---|---|
Health Review
| # | Date | Day | Overall | Task |
|---|---|---|---|---|
| 1 | 2026-07-04 | day 3 | RED | Health review 1 — sprint 22 analysis |
| 2 | 2026-07-05 | day 4 | RED | Health review 2 — sprint 22 mid-sprint analysis |
| 3 | 2026-07-10 | day 9 | RED | Health review 3 — sprint 22 close analysis |
Charts
Charts generated via compass sprint charts. See Sprint health charts for what each chart measures, its purpose, and how to read it.
PRs & Commits per Day
Dual-axis bar chart. PRs (left axis) and commits (right axis) per day. A high commits-to-PR ratio may indicate scope creep.
Daily Line Churn
Lines added (green) and deleted (red) per day. Building work produces mostly additions; refactoring produces a mix. Days with no churn may indicate blockers.
PR Cycle Time
Hours from PR open to merge, one bar per PR. Long bars indicate review bottlenecks. Generated only when PR data is available.
Cumulative Stories Done
Line chart tracking stories marked DONE during the sprint. Steady upward slope is healthy; plateauing signals a stall.
Retrospective
What went well
- Merge throughput and PR cycle time were genuinely healthy all sprint (99 PRs merged, only one ever open at a time in the fleet view, cycle times mostly under 24h) — the review/merge pipeline was never the bottleneck.
- The health-review discipline worked as designed: three System 2 reviews (day 3, day 4, close) caught and named the same underlying problem consistently rather than drifting into optimism, and this closure's scope (43 stories carried, not force-closed) was a direct consequence of taking the close review's RED verdict at face value.
- Several deep, hard pieces of infrastructure landed cleanly:
ores.orgmode+ the QA Validation Runner, the org-mode literate codegen proof-of-concept onparty, and a full Currency Pairs entity shipped end to end including its own two new authoring recipes and a skill. - Root-causing over patching, twice in one sprint: the Windows dllimport failure was traced to a real architectural mismatch (Widgets-free logic in a Widgets-linked component) rather than silencing the compiler warning, and the NATS port collision was traced to the kernel's ephemeral port range rather than just restarting the service until it worked.
What hurt
- Twenty stories were simultaneously STARTED at close, more than triple this project's own RED threshold, spanning at least seven parallel entity-commissioning efforts plus codegen, Qt tooling, compass tooling, and market-data work at once. This — not any single story running long — is what drove both the RED sprint-load verdict and the RED focus-signal verdict in every health review.
- The sprint kept growing after being told it was unhealthy: 51 stories at the day-4 review, 70 by close — 19 net new stories added in the six days after two consecutive RED verdicts, the opposite of the course-correction a RED verdict should trigger.
sprint.org's own Stories table silently drifted out of sync with 8 stories' actual state (mostly under-reporting STARTED as BACKLOG) — the sprint's own summary document was, for stretches of the sprint, not a reliable source of truth for what was actually in flight.- The initial mechanical carry-to-inbox at close swept 43 stories on
the assumption that anything not force-verified was incomplete; a
post-close re-triage against real evidence found 13 of those were
already substantially or fully done — individual task/story
Now:=/=State:fields had gone stale during the sprint rather than being updated as work actually finished. Bookkeeping discipline needs to improve: update status fields the moment work changes, not in a batch reconciliation after close. - The mission as written ("continue commissioning ores.refdata
entities and correct codegen C++ drift") covered only a fraction of
what actually shipped or was attempted — an entire Market Data
epic and the Org-mode/QA-Runner epic ran through the sprint with no
line naming them, which made "is this sprint on-mission" an
unanswerable question from
sprint.orgalone.
What changed / what to do differently
- Enforce a hard WIP limit on simultaneously-STARTED stories at the point a story is picked up, not just observe it after the fact in a health review. The data this sprint is unambiguous: PR velocity was fine, story-completion slope was fine, the only broken metric was concurrency.
- Treat a RED health-review verdict as a stop-adding-scope signal, not just a stop-and-look signal. This sprint's story count going 51 → 70 after two RED reviews is the clearest single thing to fix before the next close.
- Re-sync
sprint.org's Stories table against each story's ownstory.orgstate at every health review, not just at close — the 8-row drift found this time went undetected for at least a week. - When a sprint's actual work diverges this far from its stated
mission, update the mission mid-sprint rather than letting
sprint.orgsilently stop describing what's happening. - Update a story/task's
Now:=/=State:=/=Waiting on:=/=Next:fields the moment work actually finishes, not in a batch reconciliation after close — that discipline gap is what made the post-close re-triage necessary in the first place.
Release Notes
See release_notes.org for the full narrative.