Sprint 22

Table of Contents

This page documents a sprint (Sprint 22) of ORE Studio. It captures the sprint's mission, current status, and the stories that compose it. For the surrounding context — version goals, sprint order, and product identity — see Version 0.

Mission

Continue commissioning the ores.refdata entity library and correcting codegen C++ generation drift, carrying postponed sprint 21 stories.

Status

Field Value
State DONE
Parent version Version 0
Previous Sprint 21
Start 2026-07-02
End 2026-07-10
Now Nothing.
Waiting on Nothing.
Next Open sprint 23.
Release Notes release_notes.org
Last touched 2026-07-10

Achievements

  • Shipped the Currency Pairs & Conventions reference-data entity end to end: knowledge-base foundation, domain/db/codegen model, Qt UI (list, detail, duplicate/inverted-pair rejection, Conventions cross-navigation), and a full user-manual chapter with captured screenshots.
  • Proved out the org-mode literate codegen model on party (PR #963) — a single org file now drives both C++ and SQL generation with inline custom repository methods, replacing the two-JSON-file pattern; the mechanism is now the template for future entity migrations.
  • Landed ores.orgmode, a new C++ component parsing org-mode docs and resolving id: links via the existing org-roam index, and built the QA Validation Runner on top of it — an in-app Qt panel turning a test_scenario doc into a guided, trackable test session across the multi-worktree fleet.
  • Added codegen support for view groups (Qt detail-dialog tabs), piloted on currency, replacing the single-tab-only generated dialog layout for entities with many fields.
  • Closed out a cluster of Windows/macOS CI reliability issues: the MSVC C4273 dllimport failure (via a new ores.qt.headless extraction), a stale-Qt-plugin-after-ABI-change crash, missing EXPORT macros across 20 refdata/scheduler service classes, and the macOS relevance-guard silently skipping the build matrix.
  • Fixed a real multi-tenant correctness bug: tenant-scoped variability system settings other than bootstrap_mode weren't seeded per-tenant, silently hiding gated features for every non-system tenant.
  • Found and fixed the NATS-port-inside-the-kernel-ephemeral-range bug (moved BASE_PORT_START below 32768) after it caused a real bind collision during this sprint's own manual-chapter work — .env format version bumped to 11.
  • Ran three System 2 health reviews (day 3, day 4, close) tracking the sprint honestly through two consecutive RED verdicts to a third; the close review's findings (8 stale sprint.org rows, 20 simultaneously STARTED stories, 741 commits against a 300 target) directly drove this closure's scope: 43 of 70 stories carried back to the product backlog rather than force-closed or silently dropped.

Stories

For the definitions of the themes see Themes.

Agile

Story State Start End Description
Open sprint 22 DONE 2026-07-02 2026-07-10 Scaffold sprint 22, wire manifest, move postponed stories.
Fix Windows dllimport build failure via ores.qt.headless extraction DONE 2026-07-08 2026-07-09 MSVC C4273 broke Windows CI; root cause was Widgets-free logic living inside ores.qt.api, fixed by extracting a proper ores.qt.headless component.
Investigate RegulatoryBookTypeController Windows LNK2019 error DONE 2026-07-09 2026-07-09 Root-caused a pasted Windows LNK2019; no defect found on current main, no speculative fix made.

Product

Epic: ores.refdata Commissioning

Commission all ores.refdata entities across every access layer: verify Qt UI end-to-end post-NATS migration (list window, detail window, history, delete, eventing), implement or verify shell and CLI commands (list, add, remove), document each entity in the user manual, and file backlog captures for Wt and HTTP gaps. One story per entity. Carried from sprint 21.

Story State Start End Description
Commission: currency DONE   2026-07-10 23/27 tasks done; remaining 4 abandoned (superseded by top-level shell/CLI/Wt/HTTP stories) or found already complete (badge rendering, stale bookkeeping).
Reconcile currency Qt custom features with codegen DONE 2026-07-04 2026-07-04 Triaged currency's custom Qt functionality vs. the qt codegen profile, piloted the combo-field mechanism, and produced the regenerate plan for the Qt sync task.
Commission: rounding_type DONE   2026-07-10 Aux type: verify SQL, sync all codegen profiles, verify Qt UI, document in currency chapter.
Commission: monetary_nature DONE   2026-07-10 Aux type: verify SQL, sync all codegen profiles, verify Qt UI, document in currency chapter.
Commission: currency_market_tier DONE   2026-07-10 Aux type: verify SQL, sync all codegen profiles, verify Qt UI, document in currency chapter.
Commission: party ABANDONED 2026-07-02 2026-07-10 Verify Qt; implement shell + CLI; manual; Wt/HTTP captures.
Commission: party_type DONE   2026-07-04 Verify Qt; implement shell + CLI; manual; Wt/HTTP captures.
Commission: party_status ABANDONED 2026-07-06 2026-07-10 Verify Qt; implement shell + CLI; manual; Wt/HTTP captures.
Commission: party_id_scheme ABANDONED   2026-07-10 Verify Qt; implement shell + CLI; manual; Wt/HTTP captures.
Commission: contact_type ABANDONED   2026-07-10 Verify Qt; implement shell + CLI; manual; Wt/HTTP captures.
Commission: counterparty ABANDONED   2026-07-10 3/3 codegen-hardening tasks done; Qt/shell/CLI/manual scope still open.
Commission: book DONE 2026-07-02 2026-07-10 Verify Qt; implement shell + CLI; manual; Wt/HTTP captures.
Commission: book_status DONE 2026-07-06 2026-07-10 Verify Qt; implement shell + CLI; manual; Wt/HTTP captures.
Commission: business_unit ABANDONED   2026-07-10 Verify Qt; implement shell + CLI; manual; Wt/HTTP captures.
Commission: business_centre ABANDONED   2026-07-10 Verify Qt; implement shell + CLI; manual; Wt/HTTP captures.
Commission: portfolio ABANDONED   2026-07-10 Verify Qt; implement shell + CLI; manual; Wt/HTTP captures.
Commission: purpose_type DONE 2026-07-02 2026-07-10 Verify Qt; implement shell + CLI; manual; Wt/HTTP captures.
Commission: business_day_convention_type DONE 2026-07-04 2026-07-07 Move from ores.trading to ores.refdata (zero current consumers), then commission fully to Qt; spun off from the currency pair story.
Commission: day_count_fraction_type ABANDONED   2026-07-10 Move from ores.trading to ores.refdata (update trading_swap_legs' one reference), then commission fully to Qt; spun off from the currency pair story.
Move book, book_status, and portfolio to ores.trading ABANDONED   2026-07-10 These are trading/booking structures, not static reference data; relocate full stack out of ores.refdata.
Codegen: generate per-entity NATS event-mapping registrar DONE   2026-07-08 Hand-wired application.cpp eventing registration caused a missed-wiring regression (purpose_type); generate it per-entity instead.
Regen entities with latest codegen templates DONE 2026-07-09 2026-07-10 Country regenerated and reconciled against the shared codegen templates' new capability landed via currency's commissioning; remaining entities recaptured for future pickup.
Book data model cleanup ABANDONED 2026-07-08 2026-07-10 2/9 tasks done (the 2 live bugs); add classification flags, rename ledger_ccy, investigate legal-entity/branch fields, add rates centre, analyse allowed lists, enrich manual with domain concepts remain.
Rethink synthetic reference-data generation across entities ABANDONED   2026-07-10 Currency's ad-hoc Generate feature (client-side blue-highlighted unsaved rows, per-entity hand-wiring) is half-baked and not actually save-able; design a proper cross-entity mechanism for single + batch synthetic generation before reintroducing it anywhere.

Epic: Market data generation

Build out synthetic market data generation infrastructure across the stack — SQL variability, entity models, and librarian-driven per-party configuration — starting from FX and generalising to other asset classes over time.

Story State Start End Description
Codegen SQL variability for hypertable, bi-temporal and GIST patterns; model ores.marketdata entities DONE     SQL template variability (hypertable, soft-update/soft-delete triggers, GIST exclusion) + .org models for market_series, market_observation, market_fixing; prerequisite for full-stack codegen of ores.marketdata.
Market data cleanup: retire dead duplicate tables and preserve hand-written overrides ABANDONED   2026-07-10 Drop legacy pre-codegen ores_marketdata_series/observations/fixings_tbl (superseded, zero C++ references); remove their stale validation_ignore.txt entries; preserve market_observation's hand-written repository/service overrides against codegen regeneration.
Temporal composite entity versioning DONE 2026-07-06 2026-07-07 Design doc: child saves (e.g. party_identifiers) must bump the parent's version; loading a parent as-of version N must compose children as of that version. Work out enforcement across SQL/codegen and Qt composite version/history UX. Found via party LEI/BIC identifiers not bumping party.version.
Implement temporal composite entity versioning DONE 2026-07-07 2026-07-10 Implement the target state: SQL touch-function mechanism, as-of window-join read-side composition, Qt composite history dialog. Party/counterparty are the reference case.
Full market.txt import hits duplicate observation key DONE 2026-07-04 2026-07-04 Fixed: the file genuinely repeats ~100 (date, key) pairs, not a series_key_registry bug; import_service now de-duplicates last-line-wins with a warn/error policy; unblocks the 2016-02-05 vintage import.
Configurable feed status indicator in the FX spot grid DONE 2026-07-02 2026-07-03 FX spot grid status pips use hardcoded CSS colours; replace with a configurable indicator once the right UX (badge vs. compact status dot) is settled.
GMM improvements: tidy up synthetic data generation loose ends DONE 2026-07-02 2026-07-03 Chart axis labels, per-component + combined PDF display, and stochastic process component gaps (incl. Ornstein-Uhlenbeck).
IR Rates synthetic data generation ABANDONED   2026-07-10 Short-rate stochastic process + Curve Template raw instrument generation (deposits/FRAs/swaps), tenor-collision validation, tick-batch publishing — no bootstrapping, no vol surfaces, no QuantLib.
Synthetic data librarian support: FX foundation DONE 2026-07-02 2026-07-03 Librarian gives each new party a usable synthetic data configuration; FX only for now, but db tables/publishing built generically via codegen for future asset classes.
Synthetic data collections: Basic and Realistic ABANDONED 2026-07-03 2026-07-10 Two sample FX collections seeded from real ORE reference data (2016-02-05 vintage), Market Simulator grouping by collection, feed namespacing, vintage-availability guard.
ores.shell command for market data import DONE 2026-07-04 2026-07-04 Non-interactive market data import command, reusing the existing import_service; unblocks the synthetic data collections story's vintage import task.
Move trade import orchestration to service layer ABANDONED   2026-07-10 Move ImportTradeDialog's ~300 lines of trade-import orchestration (XML parse, per-trade save, 30-way instrument dispatch) into a server-side trade_import_service + import_trades_request, so ores.shell's future trade import command doesn't duplicate it.
Currency pair support in reference data DONE 2026-07-06 2026-07-10 Knowledge-base foundation (conventions, quoting, cross rates) followed by domain/db/codegen support for currency pairs as reference data.

Tooling

Epic: Codegen

Codegen drift correction and guardrails: resolve template-production divergence, land safety and CI checks so generated code stays trustworthy. Carried from sprint 21.

Story State Start End Description
Decommission ores.codegen.table model type ABANDONED   2026-07-10 Migrate all remaining _table.org models to ores.codegen.entity; remove the table type from the facet catalogue.
Refactor ores.codegen C++ generation ABANDONED 2026-07-06 2026-07-10 Audit/resolve template-production drift; carried from sprint 21.
Decommission legacy codegen bash scripts ABANDONED   2026-07-10 Replace direct bash invocations with compass or Python codegen; remove the scripts.
Codegen CI zero-diff invariant ABANDONED   2026-07-10 CI job regenerating all registered components, failing on any diff vs HEAD.
Codegen model safety guardrails ABANDONED   2026-07-10 Runtime guards for missing component_include/component_core; block dual-template SQL generation.
Codegen unified model — Phase 1: Qt derivation ABANDONED   2026-07-10 Derive Qt fields from conventions; drop explicit NATS protocol class names from models.
Codegen unified model — Phase 3: unify temporal templates ABANDONED   2026-07-10 Merge temporal and non-temporal C++ template families into single model-controlled templates.
Codegen developer experience improvements ABANDONED   2026-07-10 Fix spurious ERRORs in –component mode; add –entity filter; add entity group / tag system for domain-scoped generation.
Introduce per-entity NATS sub-registrars DONE   2026-07-04 Replace monolithic registrar.cpp with generated per-entity sub-registrars; isolate handler compilation; unblock incremental service API migration.
Resolve codegen model unification blockers ABANDONED 2026-07-06 2026-07-10 Carried from sprint 21.
Composite child-entity and hierarchy Qt widgets for codegen DONE 2026-07-04 2026-07-10 Handcrafted HierarchyTreeWidget + child-entity table facet, paste-block-bound, so party/counterparty can retire their hand-maintained EntityDetailDialog. Overlaps with solid_dirac's Commission: party work — compare approaches before implementing.
Codegen support for view groups (Qt detail-dialog tabs) DONE 2026-07-09 2026-07-10 The pre-codegen currency dialog grouped fields into separate tabs; codegen's Qt detail-dialog template only ever generates a single tab, making entities with many fields harder to read. Add a generic view_group concept to the entity_org schema and Qt codegen templates, piloted on currency, and document the domain knowledge in the manual.

Epic: Compass

Story State Start End Description
compass generate catalogue ABANDONED   2026-07-10 Auto-regenerate catalogue/index org files by scanning tagged org files.
Fix orphan documents and tag taxonomy DONE 2026-07-06 2026-07-10 Fix orphaned docs in org-roam graph; create tag inventory; add compass tag validation.
Compass quality of life (Sprint 21) DONE   2026-07-02 Carried from sprint 21.
Add compass client start/stop subcommands DONE 2026-07-03 2026-07-03 compass client was flat (launch-only); add start/stop matching the services/env/nats subcommand convention.
Compass GitHub Actions support ABANDONED   2026-07-10 Give compass a quick overview of all GitHub Actions runs, highlight which ones are failing, and for failing runs surface the error detail via gh CLI without needing to download full logs.

Epic: Org-mode support

Add first-class org-mode support to the C++ side of the codebase — parsing docs into a structure, resolving id: links — as a shared foundation for tooling built on top of it. First two stories: ores.orgmode itself, then a Qt QA Validation Runner built on it to address the fleet's testing bottleneck (10 worktrees, one human tester, no way to track what's under test or what was found). More stories expected here over time as other consumers emerge.

Story State Start End Description
ores.orgmode: C++ org-mode parsing and org-roam link resolution DONE 2026-07-06 2026-07-07 New component: parse org-mode docs into a C++ structure, resolve id: links via the existing org-roam SQLite index. Pure C++, usable from ores.shell; ports the shape of ores.compass's Python org parser.
QA Validation Runner: in-app test tracking for the multi-worktree fleet DONE 2026-07-06 2026-07-10 Qt panel, built on ores.orgmode, turning a test_scenario org doc's checklist into a guided test session; results and story/task context rendered in-app, not a standalone JSON file.

Infrastructure

Story State Start End Description
Verify Windows and macOS CI builds ABANDONED   2026-07-10 Re-run MSVC and macOS builds; verify the rfl C1202 fix; fix or file follow-ups.
Upgrade Qt to latest supported version to fix macOS 15 AGL removal ABANDONED 2026-07-06 2026-07-10 Qt 6.8.x references -framework AGL (removed in macOS 15); the latest supported Qt drops that dependency. CI pinned to macos-14 as stopgap.
Canary CI: near-zero sccache hit rate makes every build a cold rebuild DONE 2026-07-03 2026-07-04 Cache-quota fix applied but doesn't explain the magnitude (32 files changed vs. 98.6% miss rate); running a live zero-source-diff experiment.
Fix NATS port collision with legacy env and label inconsistency DONE 2026-07-03 2026-07-03 solid_dirac's NATS ports collided with a stray legacy env's server; migrate to base-port-derived NATS ports and unify the config label to underscore form.
Fix uninitialized asset_class/series_subclass in market_series generator DONE 2026-07-03 2026-07-03 Nightly Valgrind flagged uninitialized-memory defects in market_series_mapper::map(); generate_synthetic_market_series() never set the two enum fields.
Fix tenant-scoped variability system settings: permissions and per-tenant seeding DONE   2026-07-10 IAM can't clear a tenant's bootstrap_mode (missing DB grant, silent permission-denied); tenant-scoped system settings other than bootstrap_mode aren't seeded per-tenant, so gated features silently vanish for non-system tenants.

Documentation

Story State Start End Description
Improve site documentation discoverability ABANDONED   2026-07-10 Carried from sprint 21.

Hotfixes

Story State Start End Description
Hotfix: MacOS CI relevance-guard skips matrix build silently DONE 2026-07-04 2026-07-04 Continuous MacOS workflow reports overall Success while the scheduled relevance-guard skips the build matrix, so macOS builds are not actually being verified.
Hotfix: refdata generator/repository test fixtures out of sync with DB constraints DONE 2026-07-04 2026-07-05 Root cause: 2 counterparty codegen-facet commits introduced generator values violating DB constraints and test expectations; fixed and migrated touched literals to shared change-reason constants.
Hotfix: stale Qt plugin causes crash after ores.qt.api ABI change DONE 2026-07-08 2026-07-08 CMake had no dependency edge from ores.qt.exe to the QPluginLoader-loaded plugin libs, so an ABI change could leave a stale plugin .so segfaulting on load. Cherry-picked the fix off feature/vintage-guard onto its own branch.
Fix missing ORES_REFDATA_CORE_EXPORT on refdata/scheduler service classes DONE 2026-07-09 2026-07-09 20 service classes across ores.refdata.core and ores.scheduler.core were missing their EXPORT macro, so with -fvisibility=hidden their methods were invisible outside their own .so — causing undefined-reference linker errors for any cross-library caller (e.g. ores.iam.core.tests calling party_service/counterparty_service).

Achievements

Milestones worth capturing as they happen, so the release notes at sprint close have the evidence to hand. Add a dated ** subheading per milestone, with any screenshot via [[proj:assets/images/<file>.png]].

Captures

Real captures filed during this sprint that have not been pulled into a story as tasks — listed here purely for visibility into what the sprint surfaced. Each is an inbox capture (compass capture); triage them at the health review (promote to a story, move to next, or discard).

Capture Tags Description
     

Health Review

# Date Day Overall Task
1 2026-07-04 day 3 RED Health review 1 — sprint 22 analysis
2 2026-07-05 day 4 RED Health review 2 — sprint 22 mid-sprint analysis
3 2026-07-10 day 9 RED Health review 3 — sprint 22 close analysis

Charts

Charts generated via compass sprint charts. See Sprint health charts for what each chart measures, its purpose, and how to read it.

PRs & Commits per Day

Dual-axis bar chart. PRs (left axis) and commits (right axis) per day. A high commits-to-PR ratio may indicate scope creep.

prs_commits.png

Daily Line Churn

Lines added (green) and deleted (red) per day. Building work produces mostly additions; refactoring produces a mix. Days with no churn may indicate blockers.

line_churn.png

PR Cycle Time

Hours from PR open to merge, one bar per PR. Long bars indicate review bottlenecks. Generated only when PR data is available.

pr_cycle.png

Cumulative Stories Done

Line chart tracking stories marked DONE during the sprint. Steady upward slope is healthy; plateauing signals a stall.

stories_done.png

Retrospective

What went well

  • Merge throughput and PR cycle time were genuinely healthy all sprint (99 PRs merged, only one ever open at a time in the fleet view, cycle times mostly under 24h) — the review/merge pipeline was never the bottleneck.
  • The health-review discipline worked as designed: three System 2 reviews (day 3, day 4, close) caught and named the same underlying problem consistently rather than drifting into optimism, and this closure's scope (43 stories carried, not force-closed) was a direct consequence of taking the close review's RED verdict at face value.
  • Several deep, hard pieces of infrastructure landed cleanly: ores.orgmode + the QA Validation Runner, the org-mode literate codegen proof-of-concept on party, and a full Currency Pairs entity shipped end to end including its own two new authoring recipes and a skill.
  • Root-causing over patching, twice in one sprint: the Windows dllimport failure was traced to a real architectural mismatch (Widgets-free logic in a Widgets-linked component) rather than silencing the compiler warning, and the NATS port collision was traced to the kernel's ephemeral port range rather than just restarting the service until it worked.

What hurt

  • Twenty stories were simultaneously STARTED at close, more than triple this project's own RED threshold, spanning at least seven parallel entity-commissioning efforts plus codegen, Qt tooling, compass tooling, and market-data work at once. This — not any single story running long — is what drove both the RED sprint-load verdict and the RED focus-signal verdict in every health review.
  • The sprint kept growing after being told it was unhealthy: 51 stories at the day-4 review, 70 by close — 19 net new stories added in the six days after two consecutive RED verdicts, the opposite of the course-correction a RED verdict should trigger.
  • sprint.org's own Stories table silently drifted out of sync with 8 stories' actual state (mostly under-reporting STARTED as BACKLOG) — the sprint's own summary document was, for stretches of the sprint, not a reliable source of truth for what was actually in flight.
  • The initial mechanical carry-to-inbox at close swept 43 stories on the assumption that anything not force-verified was incomplete; a post-close re-triage against real evidence found 13 of those were already substantially or fully done — individual task/story Now:=/=State: fields had gone stale during the sprint rather than being updated as work actually finished. Bookkeeping discipline needs to improve: update status fields the moment work changes, not in a batch reconciliation after close.
  • The mission as written ("continue commissioning ores.refdata entities and correct codegen C++ drift") covered only a fraction of what actually shipped or was attempted — an entire Market Data epic and the Org-mode/QA-Runner epic ran through the sprint with no line naming them, which made "is this sprint on-mission" an unanswerable question from sprint.org alone.

What changed / what to do differently

  • Enforce a hard WIP limit on simultaneously-STARTED stories at the point a story is picked up, not just observe it after the fact in a health review. The data this sprint is unambiguous: PR velocity was fine, story-completion slope was fine, the only broken metric was concurrency.
  • Treat a RED health-review verdict as a stop-adding-scope signal, not just a stop-and-look signal. This sprint's story count going 51 → 70 after two RED reviews is the clearest single thing to fix before the next close.
  • Re-sync sprint.org's Stories table against each story's own story.org state at every health review, not just at close — the 8-row drift found this time went undetected for at least a week.
  • When a sprint's actual work diverges this far from its stated mission, update the mission mid-sprint rather than letting sprint.org silently stop describing what's happening.
  • Update a story/task's Now:=/=State:=/=Waiting on:=/=Next: fields the moment work actually finishes, not in a batch reconciliation after close — that discipline gap is what made the post-close re-triage necessary in the first place.

Release Notes

See release_notes.org for the full narrative.

Emacs 29.3 (Org mode 9.6.15)