Story: Redesign ores.trading on data-oriented principles

Table of Contents

This page documents a story in Sprint 25. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

1. Goal

ORES Studio trades by hand-evolved C++ stacks that the modeling orgs do not express. Five instrument families (bond, commodity, composite with its composite_legs child table, credit, scripted), the shared legs table (ores_trading_swap_legs_tbl), the five-group trade domain, and a wire regime that was consolidated by hand for fourteen families all live only in the tree. The consequence is that ores.trading is the one component that still cannot join the drift gate, and any regeneration the org grammar does drive silently flattens or drops tree semantics (nested domain groups, non-persisted members, bespoke service methods, hand-picked column names, the trade schema's hand-added indexes).

The story moves ores.trading to a fully modeled, data-oriented shape in the relational reading of the DOD book: no object graph is modelled, every persisted family is expressed in the modeling orgs as atomised entities with keys and foreign keys, data is normalised (sparse tables split into keyed child rows), the org grammar grows to carry nested domain groups, column-name mapping, custom service methods and wire consolidation, versioning stays a per-table property of the SCD2 spine, and the service layer is aligned to the atomised model. A drift gate then proves regeneration identity for the whole component.

2. Status

Field Value
State ABANDONED
Parent sprint Sprint 25
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-09-23

3. Acceptance

  • Every persisted ores.trading family has an org model: the 25 bound instruments, bond/commodity/composite/composite_leg/credit/scripted, the swap_legs table, trade with its identifiers. No domain class, SQL table, or wire artefact lives only in the tree.
  • All trading data is normalised and no object graph is modelled: entities are atomised rows with keys and foreign keys; tables are not sparse, and optional or repeating structure is declared as keyed child rows, not nullable feature columns.
  • The org grammar expresses nested domain groups with domain-only members, column-name mapping, org-declared custom service methods, and per-entity wire emission control. Wire consolidation is a model decision, not a tree-side deletion.
  • Regenerating trading (C++ and the trading SQL under ores.sql) is drift-clean. Trading joins the drift-check registry, and the Sep-8 drift inventory is reconciled.
  • Composite and versioning semantics survive atomisation: versioned temporal tables with the audit spine, composite as-of reads as temporal joins over keys, trade lifecycle rows and the successor chain are first-class in the model, not hand-written SQL. No aggregate or composition invariant exists.
  • The service layer exposes the batched, transactional surface (legs batch access, multi-table saves, temporal as-of reads) as generated code with no hand-restored exclusions. Multi-entity saves travel as parts messages or as container messages composed from the generated part types; containers are handcrafted first and code-generated when the grammar can express them.
  • Eventing integration tests are enabled for every instrument family through the trading-instrument profile, not per-org flags.

4. Tasks

Task State Start End Description
Analyse the ores.trading data model against the data-oriented criteria DONE 2026-09-08 2026-09-08 Run the compass-code-review-data-model review over the entire ores.trading data model: classify the component into the design regimes, walk the decision and invariant checks per entity family, and produce the in-depth analysis of current state, the divergences and their causes, and the proposed data-oriented target model with the service-layer shape changes it requires. Deliverable is the analysis document only; implementation PRs follow.
Analyse the bond family and propose its relational data model DONE 2026-09-08 2026-09-08 Bond is the pilot for the relational target of the trading analysis (dfe15809): survey every bond-related type and subtype in the tree (domain, SQL, mappers, repositories, services, messaging, the ores.ore mapping, ores.qt), then propose the entity-relationship model based on the existing tables but in the clean DOP shape - atomised entities with keys and foreign keys, normalised, no object graph. Deliverable is the analysis and ER proposal only; codegen of the types and SQL, the ore mapping update and the qt impedance bridge are follow-up tasks.
Implement the bond relational model: codegen, SQL and the loader exercise DONE 2026-09-09 2026-09-10 Bond is the pilot of the data-oriented redesign (dfe15809). Author the modeling orgs for the nine-table relational family, generate the C++ and SQL per entity, extend the population loader with the family recipe, write and validate the migration script, run the loading exercise at 1k/10k/20k rows, prove family regeneration drift-clean, and run the shell system test for the family. The old bond shape and its hand-written stack are deleted once the generated shape is in place.
Rework the ores.ore bond mapping to the relational model DONE 2026-09-10 2026-09-10 The ores.ore bond mapping becomes a mapper per entity plus an assembler over the new entities: import reads one XML product document into an issue (found or created), an instrument row and its fact rows; export joins the rows back into the product document. Flattening stops; the covered products roundtrip at the new shapes, validated through the mapper and golden suites and the ores.cli file roundtrip.
Build the ores.qt bond bridge over the relational model ABANDONED   2026-09-23 Build the ores.qt bridge over the relational bond model: the container over the generated part types, handcrafted first per the parent analysis, the qt views over the new row shapes, the wire protocol and the reworked BondInstrumentForm.
Bring the bond mapper roundtrip to zero loss DONE 2026-09-10 2026-09-12 No bond document round trips at zero loss on the mapper path: 5183 pairs are lost under the bond products and the trade envelope (3404 with a source-stated value, 1779 where the source stated an empty element) plus 47 unexplained. Fix both classes in the hand-written container and mappers, make the diff check product-aware, and deliver the shared schedule and underlyings tables so the database can hold what the container carries, per the user's answer to scope.
Add the ORE import and export verbs to the shell DONE 2026-09-11 2026-09-11 None of the three steps of the XML to database to XML path has a shell verb today: upload the import tarball to the ore-imports bucket, start the ore_import_workflow via workflow.v1.ore.import, and reconstruct the portfolio XML from trading.v1.trades.portfolio.export items. Add the verbs so the full round trip test drives the service layer through ores.shell rather than ores.cli.
Add the ORE full round trip test DONE 2026-09-11 2026-09-12 A full round trip test driven entirely through the service layer via ores.shell commands: take an ORE example document, upload it, import it to the database, export it back to XML, and classify the differences with the project's own diff script. The bond products and the envelope must read zero. Runs only after the mapper path losses are fixed, per the user's ordering.
Align bond_issue with the ORE data model DONE 2026-09-12 2026-09-12 The bond_issue entity carries only fields the ORE schema states, and every column an absent element leaves NULL rather than a sentinel, so a bond document that states only its ISIN imports and round trips.
Fix silent instrument loss from untagged variant serialisation DONE 2026-09-12 2026-09-12 reflect-cpp writes a std::variant untagged and reads it back by trying the alternatives in declaration order, so std::monostate – the first alternative of ores.trading.domain.trade_instrument – parsed from every payload. Every instrument decoded as monostate and was dropped, on both the NATS wire and the storage blob.
Retire the rfl complexity flags and close the variant traps ABANDONED   2026-09-23 The -fbracket-depth=1024 and /constexpr:depth100000 flags on ores.trading.api, and the comment naming rfl::AddTagsToVariants, were added for a variant-tagging change that was reverted. Measured: every TU on the export path compiles at bracket depth 128, so the flags are now unneeded there. Also records two traps: get_trade_instrument_response.instrument is still a std::variant, and bond_instrument_data reflects 28 fields, above the 19 that already tripped MSVC C1202 in sprint 19.
Keep stated but default-valued elements in the bond export ABANDONED   2026-09-23 A bond future states SettlementDirty, FairPrice and ExpiryLag, each with a value equal to its type default, and the export drops all three. The reverse mapper tests the value instead of the presence, so the container needs a presence flag per member. That is a modeling and codegen change to ores.trading.bond_future, not a mapper edit.
Split the bond security from the trade, following ORE ABANDONED   2026-09-23 Bond static data — the issue, its legs, and a future's contract terms — is stored once per trade because every bond table hangs off instrument_id, which carries trade_id; ORE holds the same data once per security in a reference datum.
Collapse the instrument identity into the trade ABANDONED   2026-09-23 A trade and its instrument point at each other through two nullable soft links with no referential integrity, where ORE has a single trade id; the third identity is why fields drift between the trade and the security.
Document the product identity scheme DONE 2026-09-12 2026-09-16 There is no document explaining the trade envelope against the product, the trade type catalogue is behind v17, and the product index links through a hardcoded host address.
Add a product short code to trade type ABANDONED   2026-09-23 Add a short mnemonic code to trade_type, mint one for each of the 119 product types, and correct the identifiers table on every product knowledge page.
Mark generated C++ files as generated DONE 2026-09-17 2026-09-17 Generated C++ carries no marker where the generated SQL names its template, so nothing tells a reader that a file is codegen output and a generated file gets hand-edited.
Make the trade entity regenerable DONE 2026-09-17 2026-09-23 Regenerating trade clobbers 28 of its 30 outputs because the org grammar cannot express the five-group composition, the bespoke service and handler, the hand-added SQL indexes and temporal trigger, or the Qt estate.
Regenerate the trade service layer and Qt estate DONE 2026-09-17 2026-09-18 The handler, service, repository, mapper, protocol and the nine Qt Trade files still live only in the tree, so regenerating trade clobbers them. The protocol diverges from the generated shape rather than extending it.
Regenerate the trade Qt estate ABANDONED 2026-09-18 2026-09-18 The nine ores.qt Trade files are still hand-written, and four of them read execution_timestamp as a string.

5. Decisions

  • What the nine tables cannot store, the document keeps. The exercise date list, the funding leg's schedule, the repo leg's dates and the leg's legDataType rate group have no column in the bond family tables. The mapper's container carries each one whole and export re-emits it, rather than flattening it into the issue row. The shared instrument-keyed schedule and underlyings tables of this story are their destination. Recorded at task 212862C4, waves 2.2 and 2.5.
  • BondPosition is the bond family's uncovered product, by evidence. The ORE schema declares no positionData structure, so a BondPosition has no product document to import. It has no fact table and no mapper, and the suite asserts the absence.
  • The domain path round trips at zero loss; the mapper path does not. Over the 132 example documents, save_data(load_data(x)) loses no pair, with 543 numeric and 25 boolean spelling differences only. Over the 543 documents the mapper path writes, 115572 pairs are lost and 7386 are unexplained. The difference is what each path re-emits: the domain path re-emits the document it read, and the mapper path re-emits what the rows and the container hold.
  • The trade envelope belongs to the trade entity. Trade/Envelope, CounterParty, AdditionalFields, party_id and valuation_date are lost on the mapper path throughout the tree, 2080 pairs in the bond documents alone. This story's trade work owns them, not the instrument family.
  • The three ORE path verbs are one menu, and the request id is an input. The shell's ore menu holds upload, import and export, all registered from trading_commands.cpp. The import verb waits for the workflow it starts, because the handler replies with the instance id before the work runs and the .ores language cannot capture it. For the same reason upload takes the request id as --request-id and prints it, rather than generating one the script cannot read back. Recorded at task EE79ACE9.

6. Out of scope

  • Numerical-kernel (regime 1) work: valuation and pricing logic lives in the marketdata/analytics/compute units; trading only consumes their results.
  • The models of other components (the shared ores.dq audit group, ores.refdata soft-FK parents such as book and portfolio), except for the new org keys the trading models need to express them.
  • Qt presentation features beyond what the domain and wire changes force. The trade Qt controllers' hand-evolved features are recorded in the analysis as drift evidence, not re-implemented here.

7. Result

Abandoned at sprint 25 close. Sprint 26 replaces the codegen approaches tried so far with one method: for each module, sync codegen and clear its drift. Sprint 26 opens new, clean stories for each module. This story is not carried forward.

Emacs 29.3 (Org mode 9.6.15)