Task: Implement the bond relational model: codegen, SQL and the loader exercise
Table of Contents
This page documents a task in the Redesign ores.trading on data-oriented principles story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Bond is the pilot for the relational target model of the trading analysis (dfe15809), approved and merged at PR #2039. This task implements the target model in the tree with full coverage, from SQL and codegen onwards: every family entity (bond_issue, bond_instrument, the five product fact tables, the two issue-keyed child tables) is expressed in a modeling org and generated by codegen, the hand-written bond stack is deleted once its generated replacement is in place, the migration script converts an old-shape database to the new rows, the population loader loads the family coherently at the standard sizes with the timings recorded, family regeneration is drift-clean, and the shell trading verbs round trip the family live. Every table carries the SCD2 spine with the audit columns; the four approved review answers bind the shape (leaf rows keep the full spine, a product change is cancel-and-rebook, the issue's child rows are family-owned, the repo row references the issue as its collateral).
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Redesign ores.trading on data-oriented principles |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-09-10 |
3. Acceptance
- Every family entity has an org model and codegen emits its C++ and SQL: bond_issue, bond_instrument, the five per-product fact tables and the two issue-keyed child tables, all on the SCD2 spine with keys and foreign keys per the deliverable ER.
- The migration script in projects/ores.sql/migration/ converts the current single-table bond shape into the new rows (instrument plus a share of an issue plus one fact row by trade_type_code) and is validated against an old-shape database with the row counts and joins checked.
- The population loader loads the family with a coherent population at –rows 1000, 10000 and 20000; the per-table rows/sec and the aggregate are recorded against the deliverable baseline of about 2200 rows/sec, with zero unresolvable findings.
- Family regeneration is drift-clean: regenerating each of the nine entities with –diff shows zero diff.
- The shell verb round trips pass for the family against the environment: add, list, get, history and delete per bond code with DB-row and changed-event verification.
4. Plan
(Implementation strategy. Written when work starts; key decisions
are distilled into the parent story's * Decisions at close, but the
plan itself stays — it is the historical record of what we did.)
5. Notes
Wave 1.1, the XSD survey and the deletion inventory (2026-09-09).
The deliverable names the product fact tables but does not invent
their columns. Its instruction to this task reads: "The codegen task
must read the ORE XSD product structures (bondFutureData,
bondRepoData and the Ascot shape in ores.ore's domain) to fix the
fact tables' columns, which this page names but does not invent"
(design_bond_relational_model.org, section "What each surface must
change"). This note records that reading against
external/ore/xsd/instruments.xsd, and the deletion inventory of the
hand-written stack. The ER's decided columns stay primary; the XSD
structures below are the pools the org authoring (wave 1.3) picks
from. All line numbers cite instruments.xsd.
The ten product structures and their destinations in the model:
| XSD type | Line | Composition | Destination in the model |
|---|---|---|---|
bondData |
382 | IssuerId, CreditCurveId, SecurityId; SettlementDays, Calendar, IssueDate, BondNotional, Payer, SubType, LegData |
the issue entity; every product carries one. The issue's decided columns map from it and its coupon leg |
forwardBondData |
405 | bondData + SettlementData, PremiumData, LongInForward, KnockOut | instrument row only; the forward's extra terms have no fact destination (below) |
bondFutureData |
422 | ContractName, ContractNotional, LongShort, Currency, ContractMonth, DeliverableGrade, FairPrice, Settlement, SettlementDirty, RootDate, ExpiryBasis, SettlementBasis, ExpiryLag, SettlementLag, LastTradingDate, LastDeliveryDate, DeliveryBasket |
ores_trading_bond_futures_tbl; the whole body is the candidate pool ("delivery date and the facts the XSD carries"). DeliveryBasket is a list of Ids |
bondOptionData |
2273 | optionData + strikeGroup + Redemption, PriceType, KnocksOut, bondData |
ores_trading_bond_options_tbl: OptionType feeds option_type, the strike group feeds option_strike; exerciseDatesGroup feeds exercise schedule rows |
optionData |
327 | LongShort, OptionType, PayoffType, PayoffType2, Style, NoticePeriod, NoticeCalendar, NoticeConvention, MidCouponExercise, Settlement, SettlementMethod, PayOffAtExpiry, PremiumAmount, PremiumCurrency, PremiumPayDate, Premiums, ExercisePrices, ExerciseFees, ExerciseFeeSettlement*, exerciseDatesGroup, AutomaticExercise, ExerciseData, PaymentData, SettlementData |
base of the option products; the per-trade facts the option fact row can carry |
bondRepoData |
2284 | bondData + RepoData wrapping one legData |
ores_trading_bond_repos_tbl: the repo leg carries the rate, the type and the payment dates; the issue is the collateral (review answer 4) |
bondTRSData |
2338 | bondData + totalReturnData + fundingData |
ores_trading_bond_trs_tbl: the return side (2313: Payer, InitialPrice, PriceType, observation and payment terms, PaymentDates list, FXConversion, FXTerms, ScheduleData, PayBondCashFlowsImmediately) and the funding leg (fundingData at 2297 wraps one legData) |
convertibleBondData |
2346 | bondData + CallData/PutData (cbCallData) + cbConversionData + DividendProtectionData + Detachable |
conversion terms feed ores_trading_bond_issue_conversion_targets_tbl; the call and put schedules feed call-date rows |
cbCallData |
2700 | ScheduleData + dated Styles/Prices/PriceTypes/IncludeAccruals lists (value + startDate attribute) + Soft | the convertible's call or put schedule |
cbConversionData |
2981 | ScheduleData + dated ConversionRatios (float + startDate) + FixedAmountConversion (currency + dated Amounts) + Underlying |
conversion-target child rows; the ratio is the row weight |
ascotData |
2357 | convertibleBondData + optionData + ReferenceSwapData (fundingData) |
ores_trading_ascots_tbl: ascot_option_type and the conversion-option facts the XSD carries; the reference swap is a funding leg |
callableBondData |
2431 | bondData + CallData/PutData (callableBondCallData) |
the callable's call and put schedules |
callableBondCallData |
2365 | ScheduleData + dated Styles/Prices/PriceTypes/IncludeAccruals lists | call-date child rows of the callable issue |
bondBasketData |
2566 | Quantity, Identifier, Underlying (unbounded) | no trade type is seeded for a basket product (below); its underlyings are a list |
XSD structure with no destination in the decided ER shape, recorded for the wave 1.3 decision and unit 2:
- The dated call data.
callableBondCallDataandcbCallDataattach Styles, Prices, PriceTypes and IncludeAccruals per date (each a value with astartDateattribute). The ER's call-date rows carry the schedule dates; they name no price, style, type or accrual column. A call that reprices must not be flattened away: the destination decision (extra columns on the child rows, or dated lists as more rows) lands in wave 1.3 and binds unit 2. - The dated conversion ratios.
cbConversionDatacarries ConversionRatios as dated floats and a FixedAmountConversion with dated amounts. The ER draws one conversion-target row per share with the ratio as its weight. A convertible whose ratio changes over time needs a decision in wave 1.3 (one row per dated ratio, or a recorded scope limit). - The forward's extra terms.
forwardBondDataadds SettlementData, PremiumData, LongInForward and KnockOut to the bond terms. The ER draws no forward fact table. The instrument row holds the code; the terms need a recorded decision in wave 1.3 or unit 2. - The basket.
bondBasketDatacarries no bond terms at all and no trade type row is seeded for it. The basket's underlyings belong to the shared instrument-underlyings shape of the parent story, not to a family fact table.
The ten trade type codes, verified in the seed
(ores.sql/populate/trading/trading_trade_types_populate.sql, lines
173-202): Bond, ForwardBond, BondFuture, BondOption, BondRepo,
BondTRS, BondPosition, CallableBond, ConvertibleBond, Ascot. The
reshaped instrument's trade_type_code check is the membership check
against ores_trading_trade_types_tbl through
ores_trading_validate_trade_type_fn, the mechanism every generated
trading instrument uses; the ten rows seed per tenant. The XSD's
bondData is the plain bond (seeded as Bond and named BondPosition
in the same seed); note the count mismatch with the mapper's product
set is a unit 2 concern.
The deletion inventory, verified against the tree on 2026-09-09. The whole hand-written bond stack lives in five components. A repo-wide grep for the type name and the table name finds no consumer outside them, no test fixture data, and no modeling reference.
ores.trading: 20 files. api: domainbond_instrument.hpp,bond_instrument_json_io.hpp,bond_instrument_table.hpp,bond_instrument_table_io.hpp, eventingbond_instrument_changed_event.hpp, srcbond_instrument_json_io.cpp,bond_instrument_table.cpp,bond_instrument_table_io.cpp. core: messagingbond_instrument_handler.hpp, repositorybond_instrument_entity.hpp,bond_instrument_mapper.hpp,bond_instrument_repository.hpp, servicebond_instrument_service.hpp, srcregistrar_bond.cpp,bond_instrument_entity.cpp,bond_instrument_mapper.cpp,bond_instrument_repository.cpp,bond_instrument_service.cpp. service: messagingbond_instrument_event_registrar.hppandbond_instrument_event_registrar.cpp.ores.sql: three create files and two drop files (trading_bond_instruments_create.sql,trading_bond_instruments_extensions_create.sql,trading_bond_instruments_notify_trigger_create.sql,trading_bond_instruments_drop.sql,trading_bond_instruments_notify_trigger_drop.sql), the threetrading_create.sql\irentries (lines 108-110), and the bond section oftrading_rls_policies_create.sql(lines 151-170: RLS enable plus the tenant and party isolation policies). The RLS deletion is partial: the file stays for the other trading tables.ores.qt:BondInstrumentForm.hpp,BondInstrumentForm.cpp,BondInstrumentForm.ui.ores.shell:bond_instrument_commands.hppandbond_instrument_commands.cpp.ores.ore:bond_instrument_mapper.hppandbond_instrument_mapper.cpp(the flattening mapper), plus the four roundtrip suitesxml_bond_golden_roundtrip_tests.cpp,xml_bond_mapper_roundtrip_tests.cpp,xml_bond_option_golden_roundtrip_tests.cppandxml_bond_option_mapper_roundtrip_tests.cpp.
Wave 1.2, the org-shape probe (2026-09-09).
Three draft orgs were authored in projects/ores.trading/modeling/
(flat trade_identifier-style, no profile): ores.trading.bond_issue
(issue-keyed, has_tenant_id + has_workspace_id),
ores.trading.bond_option (fact entity, has_tenant_id only) and
ores.trading.bond_issue_call_date (issue-keyed child, has_tenant_id
only, two-member primary key). Each was regenerated with
compass codegen entity generate <entity> --diff (temp-dir diff, no
tree writes) and the facets were inspected through scratch generation
in the compass venv. The verdicts below answer the probe questions of
the approved plan.
First, what all three emit: the full flat entity stack, unconditional (domain class, json_io, table, table_io, generator, changed event, protocol, handler, the three registrars, the presentation history mapper, repository and service) plus the four SQL files per entity (create, drop, notify trigger create and drop). No Qt files emit: the Qt archetypes are admissible for these model types, but no org in the family declares Qt surface, so the family lands no generated Qt until the qt bridge task decides it needs any.
Question 1, foreign keys at the data level. No second identity
field group is needed and the column-name mapping growth item (b) of
the gap task is not needed. issue_id and instrument_id are plain
columns: primary key members on the fact and child orgs, and a plain
FK column after the identity group on the instrument org (the
vanilla_swap_instrument-style shape, wave 1.5).
Question 2, the two-member primary key. SQL emission supports it
completely: primary key (tenant_id, issue_id, sequence_number,
valid_from, valid_to), the gist exclusion over both key members plus
the tstzrange, the version-unique index, the current-row id-unique
index, the insert trigger's version management, the delete rule and
the notify payload all iterate the full key list. The C++ repository
supports it: the entity declares both members as
sqlgen::PrimaryKey<std::string>, the keyed read, keyed remove and
read_at_version take both members as parameters, the batch remove
deletes tuple by tuple (the per-column .in() cross-product trap is
documented in the emitted comment) and read_latest orders by both
members. One mapper gap surfaced, F1 below. The org spells the second
member sequence_number where the ER row labels it sequence: every
SQL identifier emits double-quoted, so the keyword is harmless in SQL,
but the generated C++ member, JSON key and protocol field keep the
keyword-free spelling. Recorded as decided.
Question 3, profiles and features per entity kind. No profile growth is needed. The issue org carries has_tenant_id plus has_workspace_id; its SQL emits the workspace column with the live-workspace default, the workspace validation in the insert trigger and the workspace index. The fact and child orgs carry has_tenant_id only, matching the ER rows, which carry no workspace column on the five fact or the two child tables; their creates emit no workspace column and their triggers skip the workspace validation. The trading-instrument profile stays instrument-only (wave 1.5).
Question 4, which entities carry repositories, services and messaging. Every flat entity emits the full stack and "tables and domain only" is inexpressible without a suppression key (the recorded gap-(d) class). Decision: the generated stacks for the facts and children stay in the tree. They are the house pattern and deletion would re-drift on the next regeneration.
Finding F1, the mapper drops integer primary-key members. The mapper
template's primary-key blocks branch on is_text and is_uuid only,
so the integer sequence_number member is silently dropped in both
map directions (never read from the entity, never written to it); a
uuid or text second member maps fine (issue_id maps). The fix is a
minimal mapper-template growth item, an is_int branch at each
primary-key mapping site in
cpp_domain_type_mapper.cpp.mustache; the entity-side string-key
convention is unaffected. The child tables need this growth before
they generate for real in wave 1.3. One runtime question stays open
to the first live load: the repository compares the integer column to
string key parameters exactly as uuid keys compare today, and sqlgen's
parameter casting decides whether that binds; if PostgreSQL rejects
the comparison, the wave records the fix with the load evidence.
Observation F2, composite-key changed events. The child's SQL notify
payload arrays both key members (entity_ids holds issue_id and
sequence_number) while the generated changed event carries one id
vector (call_date_ids). No consumer of the child's events exists
yet; the composite-key eventing representation is a wave 1.3
decision.
The three orgs stay in the modeling tree. They are the wave 1.3 entities, not parked probes, so discovery picking them up is intentional; this wave writes nothing and full trading regeneration does not run again until wave 1.3 generates deliberately.
Wave 1.3, the additive entities (2026-09-09).
Five new orgs join the three probe orgs in
projects/ores.trading/modeling/: ores.trading.bond_future,
ores.trading.bond_trs, ores.trading.bond_repo, ores.trading.ascot
and ores.trading.bond_issue_conversion_target, all in the flat
trade_identifier style the probes validated. The three probe orgs'
prose was finalised. Each column list was picked from the wave 1.1 XSD
survey against the ER's decided columns, which stay primary; the
survey recorded the XSD structures as pools, and the orgs draw
instrument_id plus the per-product term columns only, exactly as the
ER page names them. bond_issue_conversion_target and
bond_issue_call_date carry the two-member key with the integer
sequence_number member; the other six carry single-member keys.
F1 closed. The mapper growth item landed as a codegen core flag plus
template branches. core.py enriches every primary-key field with
is_int when the drawer type is integer or int. Each of the four
primary-key mapping sites of cpp_domain_type_mapper.cpp.mustache
(identity and flat, entity to domain and domain to entity) gained an
is_int branch between the is_uuid branch and the section close;
the entity-to-domain casts use boost::lexical_cast<int> and the
domain-to-entity casts assign std::to_string(...) to the entity's
string-typed key member, keeping the uniform
sqlgen::PrimaryKey<std::string> convention.
The first full build surfaced a second F1-class site in the synthetic
generator: cpp_domain_type_generator.cpp.mustache assigns a
faker string noun to every composite-key extra column, which cannot
compile against the integer sequence_number domain member. The
extra-column block gained the is_int branch the natural-keys block
already had (default value zero, generator_expr honoured), with the
string fallback guarded by ^is_int. Both growth items live in the
tangled templates; the org sources carry the durable edits, per the
topology note below.
The second full build surfaced a third F1-class site in the service:
the save preconditions of cpp_service.cpp.mustache emit an
empty() check on every non-uuid key member, which cannot compile
against the integer sequence_number domain member. The per-key
validation now skips is_int members: an integer key member has no
empty state, so no empty-key guard is emitted for it (uuid members
keep the is_nil() check). The durable edit lives in
ores.cpp.service.service_impl.org, which records the validation
policy in its prose.
The tangle topology shaped how the fix landed. The .mustache is
itself generated: compass build --direct codegen_templates tangles
ores.cpp.repository.mapper_impl.org (its
#+begin_src mustache :tangle cpp_domain_type_mapper.cpp.mustache
block) and clobbers direct .mustache edits. The branches were first
edited into the .mustache and then wiped by the next tangle, which
surfaced the topology. The durable edits now live in the org source;
the tangle reproduces them. The lesson is recorded as a project
memory (tangle-codegen-templates-edit-the-org-source).
Only the two child orgs carry integer keys repo-wide, so the branch
is inert for every pre-existing entity: a python scan over the codegen
property drawers found no other primary-key member whose drawer holds
:primary_key: true and :type: integer. The registered-component
drift regeneration confirmed it: the --all run over the
KNOWN_DRIFT_FREE registry (refdata, reporting, marketdata,
compute-cpp, iam, iam-cpp, synthetic) rewrote every model and left
zero changes under those paths.
All eight entities generated for real into the tree
(compass codegen entity generate <entity>, full stack per entity:
domain, json_io, table, table_io, generator, changed event, protocol,
handler, the three registrars, presentation history mapper, entity,
mapper, repository, service, plus the four SQL files). After the
final tangle, every entity re-diffs clean: --diff reports no
differences for all eight. The component file lists were regenerated
with regenerate_cmake_component_files.py --all (three trading files
updated).
The SQL create and drop masters are wired. trading_create.sql
gains a "Bond relational model (pilot, task D7943D7E)" section with
16 ir entries: the issue create and notify trigger first, then the
two child tables, then the five fact tables. drop_trading.sql gains
the matching 16 entries in reverse: children before the issue they
belong to, the fact tables order-free, each trigger drop before its
table drop. The four hand-maintained RLS files gained the family: 8
enable-plus-policy blocks in
trading_rls_policies_create.sql and 8 drop statements in
trading_rls_policies_drop.sql.
RLS decision. The issue row carries the party scope of its family;
the fact and child rows carry no party_id and are family-owned, so
each new table gets the tenant isolation policy only. The legacy
instrument tables with party_id keep their restrictive party policy;
nothing in the family emits one. This matches the approved ER answer
that the issue's child rows are family-owned.
Observation F2 stays open and widened to a second surface. The
child tables' SQL notify payloads array both key members while the
generated changed event carries one id vector; and the generated
history-provider registrar cannot call the composite-key service
history function through the dispatch registry's single-string
history_provider interface, which the second full build surfaced as
a compile error in the child registrars. No consumer of the child
registrars exists in this wave (the hand-maintained registrar.cpp
wires legacy entities only), so the registrar template gained a
compound-key guard instead of a representation change: a single-key
entity keeps its registered provider, a compound-key entity emits an
empty registrar body with a comment naming the gap, because its rows
have no single-string id to register under. The durable edit lives in
ores.cpp.history-provider-registrar.history_provider_registrar_implementation.org and the sentinel re-diff shows the single-key
path byte-identical. The F2 representation decision lands with the
first consumer, in the loader wave or the reshape. The F1 runtime
question also stays open: whether the repository's integer-column
comparison against string key parameters binds in PostgreSQL is
settled by the first live load.
PR #2041 review round. The review of PR #2041 returned no blocking findings. Two observations were recorded instead of changed. Neither blocks the additive wave and neither is a safe org-level edit.
Child ordinal generation. The api synthetic generator emits a
hard-coded 0 for sequence_number on both child entities: no member
generator block exists for the member, so the template falls back to
its int default. No collision is reachable through the generator
today because every generated child draws its own fresh random
issue_id. A representative ordinal needs parent-keyed generation (a
pinned or parameterised parent issue_id, then a per-issue ordinal),
which is synthetic-generation design beyond a member expression. The
wave that first writes C++ fixtures holding several child rows of one
issue picks this up.
Default construction of the children. sequence_number has no
in-class default while version does. The :default_value:
property could express a 0 default, but defaulting the int alone
leaves a default-constructed row unreadable: issue_id stays
uninitialized like every uuid primary-key member in the tree. A
deterministic default for all primary-key members is a
domain-template decision; it rides with the F1 follow-up work.
Wave 1.5, the reshape, part 1: the mapper switch-over decisions (2026-09-09).
The ores.ore mapper switch-over is pulled into this wave by the approved plan: the mapper roundtrip suites compile against the mapper, so the mapper must target the new rows or the wave loses coverage. The deep rework (issue find-or-create, per-entity mappers and the assembler, full ten-code coverage) is unit 2 of the mapping task. This note records the boundary between the two and the container decisions the wave implements.
The variant carrier. The bond alternative of trade_instrument (the
only in-memory carrier between import and export) becomes an assembled
container over the generated part types, handcrafted per the plan's
unit-3 instruction. It mirrors the rates precedent shape
(swap_instrument_data = with_legs<rates_instrument_variant,
swap_leg>) with named rows instead of a leg vector: the slim
bond_instrument header row, its bond_issue row, the engaged
per-product fact row (bond_option, bond_trs or bond_repo) and a
document remainder holding the fields reverse needs that no row yet
carries. The remainder exists because the option exercise dates have
no destination in the nine tables (recorded scope limit, wave 1.3):
reverse_bond_option emits the exercise date today, so dropping it
would change element presence, which the wave's fidelity definition
forbids. The remainder rides the container until the shared
instrument-keyed schedule tables of the parent story give the dates a
row.
Forward storage, reverse emission. The wave's fidelity definition:
reverse emits exactly what it emits today for the seven mapped
products; forward storage retargets to the new rows. The forward
mappers fill the issue row with the fields map_bond_data filled into
the wide terms and features (settlement_days is an issue column), the
slim header with trade_type_code and issue_id, and the fact row of
the product with its economics. BondOption stores option_type and
option_strike in bond_option; BondTRS stores return_type
"TotalReturn" (the hardcoding stays, unit 2 removes it) and the funding
leg type and index in bond_trs; BondRepo stores the repo leg type
and rate in bond_repo, new capture. The coupon-frequency corruption
in forward_bond_repo ("Quarterly"/"Maturity" overwriting the issue's
real tenor) disappears: a shared issue row cannot carry a per-trade
artifact. Where the collateral carries a real schedule the output
keeps its real tenor, a value-only change; where it carries no leg
data the fabricated Tenor element vanishes from the output, which
converges toward the upstream document the roundtrip check compares
against. The element inventory therefore improves or stays neutral on
every repo file. The container mints
one fresh issue_id per mapped trade and pins instrument.issue_id
to it; deduplication over security_id is the DB boundary of unit 2.
The DB-bound import boundary. The workflow import handler
(ore_import_execute_handler.cpp) saves the slim instrument header
row only for the bond family in this wave, mirroring the rates
precedent, whose DB import persists header rows and drops the legs.
Issue find-or-create, the saving of issue and fact rows, and the
dedup over security_id land in unit 2 ("Import reads one XML product
document into an issue (found or created)"). The wave records that the
workflow-imported bond rows therefore carry no readable economics
until unit 2: the instrument header persists, the issue it points to
does not.
The classification visitor catch. The wave's ctest run caught the
one silent casualty of the switch-over. importer.cpp stamps the
routed product_type by type-testing each instrument-variant
member, and in the reshape the bond branch of that visitor went
dead, so bonds imported with an empty product_type
(xml_trade_import_tests.cpp:343 failed on the empty value). The
fix binds the branch to the generated bond_instrument_data member.
An if constexpr visitor branch that matches no live member
compiles clean and eats its input silently; only the suite's
classification assertion trips on it. The assertion is the tripwire
the switch-over leaned on.
Wave 1.5, the reshape, part 2: the ores.qt minimum (2026-09-09).
Decision: keep the BondInstrumentForm trio and rebind it to the assembled container. The plan's deletion scope is the hand-written bond stack the generated files supersede, the ores.trading facets (domain, table io, repository, service, messaging, registrar); the qt form is not in that set. The plan's wave sentence commits the form to compiling and working against the new rows in this wave, and the full bridge (per-product views, parts save) is the qt bridge task's unit. Deleting the trio would trip the registry's no-form-registered warning on every bond open until that unit lands. The rates precedent (SwapInstrumentForm) shows the house pattern: the form binds the wire shape and persists what the wave's save handlers persist.
What the minimum changes:
- The form binds the container:
data_(bond_instrument_data) replacesinstrument_(the wide type). Itspopulateoverload takes the container, matching the base class, whose bond overload already moved to the container in this wave. - The widget mapping follows the row ownership: the economics and
optional tabs bind the issue row (issuer, currency, face value,
coupon rate, coupon frequency, day count, issue and maturity dates,
settlement days and description all live on
bond_issue); option type and strike bind the engagedbond_optionfact; the TRS return type and funding index bind the engagedbond_trsfact; option expiry binds the container's document remainder, which is where reverse_bond_option keeps the exercise date. - Four widgets are removed with the facts they edited, whose rows the
container does not carry in this wave: call date (child rows),
conversion ratio (child rows), future expiry date (
bond_future) and the ascot option type (ascots). Keeping them bound to nothing would silently eat input; the bridge's per-product views add the child-row tables. - The save path sends the slim header row only
(
req.data = data_.instrument), mirroring this wave's DB import boundary. The write direction engages a fact row only when its combo carries a value, so an empty extensions page cannot fabricate a fact for a plain bond. - The two GUI import call sites (OreImporter and ImportTradeDialog)
had silently dead bond branches: they matched the wide type, which
is no longer a
trade_instrumentalternative, so imported bond trades saved no instrument at all. They now match the container and save its slim row.
Recorded limitation, binding unit 3: the form persists the slim row
only; issue and fact edits ride the in-memory container; the GUI
create path cannot satisfy the issue_id FK in this wave; and loaded
rows display what the service assembles, which is the slim row until
unit 2 persists and serves the issue and fact rows.
Wave 1.5, the reshape, part 3: the ores.shell verb (2026-09-09).
Decision: the verb add persists the assembled parts; it does not
mirror the slim-only save of the DB import and the qt minimum. The
plan's wave sentence says the verb changes "what add constructs and
what get returns: the assembled instrument with its issue and fact
rows". The wave's part 1 and part 2 records bind slim-only persistence
to surfaces a later unit reworks: the DB import boundary is unit 2's
("Import reads one XML product document into an issue (found or
created)") and the qt parts save is unit 3's. No task owns the shell
verb after this wave, yet the wave-1.7 acceptance runs live shell
round trips with DB-row verification inside this task. A slim-only
shell add would insert instrument rows whose issue_id names no
stored issue (the SQL has no REFERENCES constraint; the FK is
model-level), a state no later wave repairs. The verb therefore
follows the rates precedent, where the add constructs what its rows
carry, and persists in order through the generated per-entity save
services: one fresh issue row (minted issue_id) holding the bond
terms, the slim instrument row pinned to that issue, and the engaged
fact row of the product (bond_option for option_type, bond_trs
for trs_return_type, whose funding index takes
trs_funding_leg_code, mirroring the qt minimum). The three saves are
not atomic; the failure report names the surviving row ids. That is
the recorded price.
The four dropped add arguments. The call date, future expiry date,
option expiry date and the ascot option type leave the add surface:
their wide columns have no row in this wave, and the qt orphan rule
applies (a bound-to-nothing argument would silently eat input). Their
destinations are the child tables, bond_future, the shared schedule
tables and ascots of later units. The verb also refuses to combine
the option and trs fact arguments on one add: the product carries one
fact row.
What get returns. List, get and history return what the generated
service serves: the slim instrument rows, whose table io renders the
identity, the issue_id and the audit tail. The response member
rename (instruments to bond_instruments) rides the regenerated
protocol. Engaged facts write on add and the issue row persists on
add; the read surfaces of the fact rows are unit 2's and unit 3's.
Wave 1.5, the reshape, part 4: the deletion inventory outcome (2026-09-09).
The wave-1.1 inventory named a 20-file ores.trading stack, five ores.sql files and the RLS section for deletion. The regeneration ran over the same paths, so the outcome differs per path kind.
- The instrument stack paths regenerate: the org emits every facet the hand-written stack had (domain, json_io, table, table_io, changed event, handler, repository entity, mapper and repository, service, registrar) at the same paths, so the regen overwrote the hand-written files in place. The pure additions the wide shape never had (the per-entity registrar, the per-entity protocol, the generator and the presentation history mapper) land as new files.
- Two inventory paths came out byte-identical: the service event
registrar pair and the api
json_io.cpp(its TU carries no shape, it includes the header). Nothing to delete there; the regen already produced that content. - The ores.sql side: the extensions create file is deleted (the wide
extension columns have no table in the reshaped model), the two
trading_create.sql\irentries and the drop references for it are gone, and the remaining four files regenerated at their paths. The RLS section keeps its partial deletion: the file stays for the other trading tables, and noextensionsreference remains anywhere inprojects/ores.sql. - The qt trio, the shell verb pair and the ores.ore mapper are not deletion targets: parts 2 and 3 above rebind them to the new rows, and the four mapper roundtrip suites stay with the mapper.
Net deletions: one SQL file. The hand-written instrument stack exists in the tree only as the regeneration's input.
Wave 1.4, the loader extension (2026-09-09).
scripts/load_trade_population.py now loads the family to its model
premise. The ten trade-type codes cycle over the bond instrument rows,
so block j of ten consecutive rows trades one ISIN. The issue row of
that ISIN, the issue-keyed child rows and the per-product fact rows
load one per block. The family tables therefore load at rows / 10;
every other table loads at rows. The family set derives from the
loader's own table constants, and the header prints the family count
when every family table is present.
The chains the wave names are wired at the seed level, per the
loader's uuid5 grammar. The instrument rows of code k sit at the
row indices congruent to k modulo ten, so the j-th fact row of a
product pairs instrument row 10j + code_index by seed; a
recomputation of the expected seed set per code over the live rows
proves the layout (all ten codes, zero mismatches). The child rows
couple to the issue rows row-wise through the new REFERENCES
entries, one child row per issue. The instrument.issue_id hook
fires only when the reshaped instrument table replaces the legacy one
in wave 1.5; the legacy table has no such column, and the comment on
the hook records that.
Account validation. The insert triggers validate modified_by and
performed_by through ores_iam_validate_account_username_fn, which
is strict whenever accounts exist. This schema-era validation postdates
the deliverable's baseline loader runs, which is why the first probe
of this wave loaded zero tables with username findings. The loader now
resolves a service account up front (ores_brave_hopper_analytics_service
on this database) and reuses it for both columns. The enumeration
learning count is unchanged at 23.
Loader evidence. The gate run at --rows 1000 --keep loaded 49
tables and 41,800 rows in 10.88s, with the family at 100 and zero
findings. Coherence was verified by SQL over the kept population as
the loader's own user: per-table counts exact (instruments 1000,
issues, children and facts 100 each), the ten codes at 100 rows each,
100 distinct XS-shaped security_id=s, zero orphaned children, zero
orphan, wrong-code or uncovered fact rows per product, and no NULL in
any family column. The database holds the kept population for wave
1.5's use. The deterministic seeds mean a =--keep rerun over a
standing population collides on primary keys; a default (non---keep)
run first removes the bench rows, which is the loader's own cleanup
path.
RLS measurement lesson. The loader reaches postgres as the .env
PGUSER (the postgres superuser here), so COPY bypasses row-level
security on every table, the RLS-enabled legacy tables included. A
verification session as the DDL user is RLS-filtered instead: the
legacy tables' party-scoped policies hide their rows when
app.current_party_id is not set, while the family's tenant-only
policies let rows through under the default system tenant. A first
verification pass misread those filtered counts as vanished rows;
counts for loader verification must run as the loader's own user. The
ownership split is recorded too: the family tables are owned by the
test DDL user (hand-applied in wave 1.3) and the legacy tables by the
main DDL user. The loader is unaffected, and the wave 1.5 recreate
resolves the split.
The wave leaves observation F2 open: the loader copies rows and never consumes the generated registrars or repositories, so the compound-key registrar representation and the F1 repository-bind question both land with the first real consumer in the reshape.
Wave 1.5, the reshape, part 5: the migration script (2026-09-09).
projects/ores.sql/migration/trading_bond_instruments_flat_columns_to_relational_rows_migration.sql,
styled on the synthetic reference: one-shot, single transaction,
applied by hand. The docstring orders the work in three steps. The
eight family create scripts apply first, with their tables holding
no rows for the migrating tenants. The migration runs second. The
reshaped trading_bond_instruments_create.sql and
trading_bond_instruments_notify_trigger_create.sql run last.
Until they run, any write to the instrument table fails loudly: the
legacy insert trigger and notify trigger stay installed, and their
function bodies reference the pre-reshape id column. On a freshly
recreated database the instrument table has no id column and the
script is a guarded no-op.
The guards. Three, all before any mutation. The legacy id column
must exist, else the script notices and returns: this is the
idempotency guard and the ordering guard in one. Every
trade_type_code must sit inside the ten, else the script aborts.
The eight family tables must hold no rows of the migrating tenants,
else the script aborts naming the table. The loader population on a
dev database trips the third guard, which is why validation runs on
a wiped scratch copy.
The only join. The wide table never carried a security identifier,
so the migration derives one per row: 'MIG-' || md5 over the eight
NOT NULL term columns, computed once in a snapshot temp table.
Merging is safe by construction: a group is exactly the rows that
agree on every fingerprinted term. The fingerprint excludes
settlement_days and description, so a group can disagree on
them; the issue row takes the earliest row's values, and disagreeing
groups and null settlement_days (written as 0) are counted and
noticed. The deliverable's recorded limitation applies: the wide
table carried no identifier to tell two different securities with
identical terms apart. The minting inserts group by (tenant_id, fp):
a select distinct over gen_random_uuid() would defeat its own
dedup, because the uuid differs per row. Current rows mint open
issues (valid_to = infinity) with the group's earliest valid_from.
Closed rows whose fp matches a live issue join it; the rest mint
closed issues from their own terms, one per fp, carrying the
validity window of the latest row that bears the fp. Every
instrument row maps 1:1 to an issue; the migration counts orphans
and aborts if any exist.
The surgery. In place on the instrument table: rename id to
instrument_id (the rename cascades to the primary key, the gist
exclusion, the checks and the indexes), add issue_id, fill it from
the map, set it NOT NULL, drop the 19 moved columns with their
inline checks, and add the ten-code check explicitly. The table's
triggers disable around the migration's own UPDATE, because the
notify trigger fires on UPDATE and its installed body references the
pre-reshape column.
The raw inserts. Issue, fact and child rows insert with the target
insert triggers disabled, and the audit columns are set by hand. The
issues insert trigger validates the account username, which raises
for a DBA session user, and rewrites valid_from and valid_to on
every insert; the fact and child insert triggers validate the
account username too. The minted issue rows must bypass the issues
trigger for the same reason as the raw fact rows. A read of the
trigger bodies before the first validation run caught the minting
section arming the issues trigger: the run would have raised on the
session user and, under a validated account, would have rewritten
the windows the migration had just chosen. The disable moved ahead
of the mints. New rows are version 1 with valid_to equal to
infinity and valid_from equal to the source row's valid_from
for facts and children. Every legacy family row is single-row, so
version-1-with-full-window is exact and amended SCD2 history is not
preserved because none exists.
The audit. New rows take change_reason_code 'system.new_record'
with the commentary 'Migrated from the wide legacy bond instrument
columns'. The reference migration used 'system.data_migration', but
that code is not in the canonical change-reason catalogue, which
holds 'system.new_record' and 'system.test' only. modified_by =
coalesce(ores_iam_current_service_fn(), current_user) and
performed_by = current_user, as in the reference. Instrument rows
keep their own audit and validity unchanged.
The gates. Fact and child rows migrate by code. Options take
option_type in (Call, Put) with option_strike at least 0. TRS
rows take TotalReturn or PriceReturn with a funding leg code, and
the code decodes: 'Fixed' becomes a fixed leg with no index, any
other code a floating leg carrying the code as the funding index;
funding_rate stays null, because the wide table never carried it.
Ascots take any non-null ascot_option_type, and the loader-era
junk copies faithfully: the ascot table has no check to reject it.
Call-date children come only from CallableBond rows, deduplicated
per (issue, call_date), with per-issue sequence numbers. Rows that
fail a gate are left unmigrated and counted; the failure is a
notice, never an abort. Three legacy columns have no destination,
and their values are noticed before the drop: future_expiry_date
(the futures fact table carries no such column, and the wave-1.1 XSD
survey supersedes the deliverable's mapping row),
option_expiry_date (the exercise-schedule tables are not part of
this wave) and conversion_ratio (conversion targets need an
underlying identifier the wide table never carried). BondRepo rows
do not migrate: the wide table never carried repo economics. The
script ends with a union-all summary select over the nine tables and
an integrity select counting instrument rows without an issue.
The validation vehicle. The script validates on an old-shape database: a pg_dump/pg_restore scratch copy of the dev database, taken after the ctest run so the snapshot never catches a mid-test reset. The loader-era family rows are wiped, crafted duplicate-terms and per-code rows are added, and the run asserts the row counts, the 1:1 instrument-to-issue map, the dedup join and the chosen validity windows. The guard aborts demonstrate safely on the real dev database: the guards fire before any mutation, inside the single transaction. Results land in the PR description.
Populate and fixtures. Nothing to update. The audit found no populate, testdata or fixture writer of the legacy table, and the loader needs no change for the reshape. Wave 1.6 runs it as it stands.
The validation run. Two scratch copies of the dev database carry the
run: ores_bond_mig_val for the migrate path and ores_bond_mig_guard
for the pristine guard demos. On ores_bond_mig_val the crafted set
(14 rows, ids …01-…14, fps A-K) was raw-inserted with the legacy
instrument triggers disabled, the eight family tables were wiped, and
the script ran end to end: exit 0. The summary select matched the
derived numbers exactly: instruments 1018, issues 1014 (1010 open,
1000 loader fps plus 10 crafted fps; 4 closed, 3 loader fps plus
fp_B), options 1, trs 2, ascots 101, call dates 102, futures, repos
and conversion targets 0, orphans 0. The dedup join held: the fp_A
group (T1-T3 current, T4 closed) minted one open issue carrying the
terms of its earliest row (T1, settlement 5, window
2026-01-15-to-open) with all four instruments joined to it; the
closed fp_B minted a closed issue with the bearer window
2026-08-01-to-2026-09-01 and no fact; the loader closed pair
36a7f7aa2a94 minted one closed issue on the max-valid_from bearer
window. The gates held: the option fact is T6 only (closed T5 and
strike -3 T7 refused), the trs facts are T8 (Fixed leg, null index)
and T9 (Floating leg, SONIA index). The ten-code check, the dropped
legacy extension columns and the audit values
(system.new_record / "Migrated from the wide legacy bond instrument
columns") were all asserted. Two notices exceeded the survey: null
settlement 5 and variance 2. The guard copy explains both: the four
loader closed rows carry null settlement (current rows carry 1), and
the loader's only multi-row fp group is the closed pair, which varies
in description. The notices are correct.
The run found one defect, now fixed: the plain temp table
bond_mig_issue carried on commit drop between its name and column
list, which the grammar allows only in the CTAS form used by the
other two temp tables. The clause moved after the column list and the
run went clean.
The guards demoed on ores_bond_mig_guard: guard 1 (re-run on the
migrated scratch) notices "no legacy id column" and exits 0 with the
counts unchanged; guard 2 aborts on a raw-inserted out-of-ten row
("Aborting: 1 instrument rows carry a trade_type_code outside the ten
codes.") with the id column untouched; guard 3 aborts naming the
first offending table ("Aborting: ores_trading_bond_issues_tbl holds
rows for the migrating tenants."). Each guard fired before any
mutation. The delete-rule demo note: the legacy instrument delete
rule is INSTEAD OF, so removing a demo row on the scratch needed the
rule disabled first. Results land in the PR description.
Wave 1.5, the reshape, part 6: the recreate and the verification cycle (2026-09-09).
The dev database now lives at the reshaped shape. compass db
recreate -y -k ran clean (47 s, schema version 0.0.25, git commit
cc013f1589 stamped). The two scratch validation databases dropped
first: the environment roles own objects inside them, and a role drop
refuses while such databases exist. The live schema was re-verified
after the recreate: the instrument table has its 14 columns with the
ten-code check and issue_id NOT NULL, and the nine family tables
exist under the recreated roles.
The baseline population ran on the new shape: --rows 1000 --keep
loaded 49 tables and 41,800 rows in 6.55 s with zero findings and the
23 enumeration learns unchanged. The family loads at its model
premise: instruments 1000 rows at 7123 rows/sec, and the eight family
tables 100 rows each at roughly 1400 to 2040 rows/sec. This is the
empirical proof of the populate finding in part 5: the loader needs
no change for the reshape, because it introspects the deployed
schema. Coherence of the kept population on the reshaped tables:
zero instrument rows without an issue, exactly 10 instruments per
issue, and no issue spans more than one trade-type code.
The full verification cycle is green against the recreated database.
ctest: 71 of 71 suites passed in 908.63 s, the trading, shell, cli,
ore and qt suites among them, all against the reshaped schema. The
domain roundtrip check exits 0 with the bond products at partial
fidelity, the recorded mapper state of this wave; the deep import and
export rework is unit 2's scope. Codegen drift: the template build is
clean, check_component_drift.py --all reports no drift, and
regenerate_cmake_component_files.py --all --check finds the
component lists up to date. The compiled delta of the wave was
verified green at commit 7e1d1426a0; the fix commit cc013f1589 that
follows carries only migration SQL and this record, so the full build
state carries over unchanged.
The tree sits at cc013f1589 on
feature/implement-bond-relational-codegen with a clean working
tree and nothing pushed. The validation results of this wave land in
the PR description at raise time, on user direction.
Wave 1.6, the loading exercise and the family drift gate (2026-09-10).
The database was rebuilt from HEAD before the exercise: compass db
recreate -y -k exited 0 in 48 s, schema version 0.0.25, git commit
ba0eee6234 stamped, and all nine family tables are present on the
recreated schema. Every loader run in this wave reported zero
findings and closed clean (0 marked row(s) remain, 0 rule(s) left
disabled).
The population loader ran at the three standard sizes. The same size does not repeat, so every sample is recorded.
--rows |
Tables | Rows | Seconds | Aggregate rows/sec |
|---|---|---|---|---|
| 1000 | 49 | 41,800 | 6.80 | 6,147 |
| 1000 | 49 | 41,800 | 23.47 | 1,781 |
| 10000 | 49 | 418,000 | 127.09 | 3,289 |
| 10000 | 49 | 418,000 | 202.16 | 2,068 |
| 20000 | 49 | 836,000 | 589.10 | 1,419 |
| 20000 | 49 | 836,000 | 306.06 | 2,731 |
The other family tables are small by construction: --rows 1000,
10000 and 20000 give each of them 100, 1000 and 2000 rows, and give
bond_instruments 1000, 10000 and 20000. At 100 and
1000 family rows the per-table figure is dominated by per-statement
overhead and moves by a factor of two to four between runs (1k:
979 to 2024 rows/sec, 10k: 2028 to 8182). Only the 20000 sample
carries signal, and there the two runs agree within 2% to 7%:
| Table | 20000 (a) | 20000 (b) |
|---|---|---|
| bond_instruments | 8423 | 7948 |
| bond_issues | 8864 | 9046 |
| bond_options | 10155 | 9712 |
| bond_futures | 9731 | 9061 |
| bond_trs | 10143 | 9432 |
| bond_repos | 9855 | 9558 |
| ascots | 9731 | 9401 |
| bond_issue_call_dates | 9881 | 9386 |
| bond_issue_conversion_targets | 9329 | 9491 |
Against the deliverable baseline (design.org: about 2200 rows/sec
on the deployed schema, 56032 rows/sec on a triggerless clone,
notify at 1.5%, and sizing guidance of about 2000 rows/sec for a
bulk path): the family lands at 8864 to 10155 rows/sec, about four
times the 2200 baseline and well above the guidance. The aggregate
lands on both sides of the guidance across the six samples (four
above it, two below), so the size alone does not predict which side
a run falls on. That split is not a property of the new shape: the
reshape's own table, bond_instruments, runs at 7948 to 8423
rows/sec in the full run and 8908 isolated.
The aggregate is dragged down by legacy tables. Five of them are
slow in every run and are the slowest five in three of the four
runs: vanilla_swap_instruments, party_roles,
party_role_types, trade_id_types and fpml_event_types.
activity_types usually joins them. At 20000 rows they ran at 930
to 1098 rows/sec in the first run and 365 to 598 in the second. The
attribution run isolates the contrast:
trade_typesat 20000, loaded alone: 7530 rows/sec with all its triggers, and 7149 with the notify trigger disabled. The reading moved the wrong way by 5%, so notify is not a measurable cost at this resolution, which agrees with the deliverable's 1.5%. In the first full run the same table took 20.7 s at 966 rows/sec, a 7.8x penalty against isolation.bond_instrumentsat 20000, loaded alone: 8908 rows/sec, and 8741 with the notify trigger disabled. That reading also moved the wrong way, by 1.9%. In the full run, 8423 rows/sec, about a 5% penalty.- The insert trigger cannot be disabled on either table to measure
the rest of the cost. Its version management is what keeps the
bitemporal exclusion constraint satisfied; with the trigger off
the load fails with
conflicting key value violates exclusion constraint.
So the in-run cost of a table is not a function of its own trigger pair. The slow set also moves between runs: 22 tables took over 18 s each in the first 20000 run, 6 took over 8 s in the second. The cause is not isolated in this wave. Per the plan the split is the recorded price of storing the truth, not a failure, and the diagnosis belongs with the parent analysis's performance work. This task records the dependency and does not pull it in.
The family drift gate is clean. All nine entities regenerate with
zero diff under compass codegen entity generate <entity> --diff,
each reporting "No differences." at exit 0: ascot,
bond_future, bond_instrument, bond_issue,
bond_issue_call_date, bond_issue_conversion_target,
bond_option, bond_repo and bond_trs. The accepted slugs are
the bare entity names; the dotted form (ores.trading.bond_instrument)
is not a match and reports "No entity found matching".
compass codegen entity list prints the accepted slugs.
Component-wide drift registration and the Sep-8 inventory reconciliation are not part of this gate. They belong to the port story and to the parent analysis's drift PR, per the parent story's record. The dependency is recorded here rather than pulled in.
Wave 1.7, the shell system test (2026-09-10).
The environment was rebuilt from HEAD before the test: compass db
recreate --yes exited 0 in 1 m 11 s at schema version 0.0.25 with
git commit 0fe19dc0c7 stamped, and all nine family tables are
present on the recreated schema. The fleet restarted from freshly
built binaries and all 22 units reported "Service ready". The Acme
tenant was provisioned with the standard script,
c195148f-d3aa-4cc1-9dd8-9023afee25ed with the default party
"Lloyds Wealth Management Ltd"
01a08ab2-189b-72e2-843f-8a1ad34e1955. The subscriber ran on the
prefixed subject ores.dev.brave_hopper.ores.trading.> for the
whole test, per the generic-batch protocol recorded on task C36A4ECA
of the shell-trading-management story.
The protocol runs at eleven probes, so that every one of the ten
trade type codes crosses the verb. BondTRS runs twice, once per
funding-leg branch.
| Probe | Code | Fact arguments |
|---|---|---|
| B01 | Bond | none |
| B02 | ForwardBond | none |
| B03 | BondFuture | none |
| B04 | BondOption | Call, strike 1000 |
| B05 | BondRepo | none |
| B06 | BondTRS | TotalReturn, fixed leg |
| B07 | BondTRS | PriceReturn, floating GBP-SONIA |
| B08 | BondPosition | none |
| B09 | CallableBond | none |
| B10 | ConvertibleBond | none |
| B11 | Ascot | none |
Every add printed its instrument id and issue id. The DB layer
confirms all eleven: eleven open instrument rows with the matching
trade_type_code, and eleven open issue rows joined on issue_id
carrying the correct security_id, issuer, currency, face_value,
coupon_rate, coupon_frequency_code, day_count_code,
issue_date, maturity_date, change_reason_code = 'system.test'
and the provisioned default party. The fact rows are one open option
row (Call, strike 1000) and two open TRS rows (TotalReturn with
a fixed leg and an empty index, PriceReturn with a floating leg
and GBP-SONIA).
The paginated get showed 11 of 11 with the matching type and issue
id, and history showed version 1 for all eleven. The eleven deletes
soft-closed every row: the live view returned to "Page 1 of 1 (0 of
0 total)", and the DB showed 0 open, 11 soft-closed and no new rows.
history after the deletes still shows the version 1 row for each
id, so a closed row stays readable.
The changed-event stream carried 28 deliveries on the prefixed subject, over 14 distinct instrument ids with exactly one save and one delete each:
| Deliveries | Source |
|---|---|
| 1 to 11 | the eleven probe saves |
| 12 to 22 | the eleven probe deletes |
| 23 to 25 | the three negative partial instrument saves |
| 26 to 28 | the three negative cleanup deletes |
Every payload carries the matching instrument id and tenant id. The trade-type negative emits no event, because the instrument save is the publisher and that save is the one that fails. Its issue row still lands and stays open, one of the four that the partial-state finding below counts.
The CHECK gate closed the wave: four negative adds, each tripping its own constraint, with the name read off the live schema.
| Negative | Argument | Constraint |
|---|---|---|
| N01 | trade_type_code FxOption |
ores_trading_bond_instruments_tbl_trade_type_code_check |
| N02 | option_type Straddle |
ores_trading_bond_options_tbl_option_type_check |
| N03 | return_type ExcessReturn |
ores_trading_bond_trs_tbl_return_type_check |
| N04 | option_strike -50 |
ores_trading_bond_options_tbl_option_strike_check |
Three findings came out of the run.
The shell delete does not reach the fact row. After the eleven
deletes one option row and two TRS rows stayed open, with no open
instrument joining to them. The wire protocol already carries
trading.v1.bond_options.delete, so the gap sits in the verb and
not in the service. It is recorded for the parent story's
service-layer work rather than fixed here, because the fact tables
have no service of their own until that PR.
funding_rate stores as NULL when the verb passes 0.0. The codegen
0-to-NULL sentinel is the cause: a nullable numeric column with a
non-optional C++ type maps 0 to unset on the write path and reads
back through value_or(0). The same rule governs
bond_repo.repo_rate and credit_instrument.index_series, so this
is a codegen-wide convention and not a bond defect. The verb
documents the choice at the assignment.
The add is not atomic. It writes the issue, then the instrument, then the fact row through three saves, so a later failure leaves the earlier rows. The four negatives left four issues and three instruments behind, and the cleanup deleted the three instruments. The verb names each partial state in its failure message, and row 7 of the Review table records the boundary.
The final tally on the recreated schema. Instruments 14 total and 0 open, issues 15 and 15 open, options 1, TRS 2, and repos, futures, ascots, call dates and conversion targets at 0. The fifteen issues stay open because the verb surface has no issue delete: the instrument delete soft-closes the instrument row alone.
Wave 1.7 rode two pull requests. PR #2046 went up first, against the wave 1.6 branch. Merging wave 1.6 in PR #2045 deleted that base branch, and GitHub closed #2046 with it, because a closed pull request cannot take a new base. The same commit set went up again against main as PR #2047, carrying the review fix, and that is the one that lands.
6. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario, run
through the QA Validation Runner panel) that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
7. PRs
| PR | Title |
|---|---|
| #2047 | [ores.trading] Bond relational model: the shell system test, wave 1.7 |
| #2046 | [ores.trading] Bond relational model: the shell system test, wave 1.7 |
| #2045 | [ores.trading] Bond relational model: loading exercise and family drift gate, wave 1.6 |
| #2044 | [ores.trading] Bond relational model: the reshape, wave 1.5 |
| #2042 | [ores.trading] Bond relational model: population loader wave 1.4 |
| #2041 | [ores.trading] Bond relational model: additive codegen waves 1.1 to 1.3 |
8. Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | The synthetic generators always emit 0 for sequence_number |
bond_issue_call_date.org (and bond_issue_conversion_target.org) |
Recorded, not changed | No collision reachable today: each generated child draws its own random issue_id. A real ordinal needs parent-keyed generation. See the Notes entry. |
| 2 | int sequence_number has no in-class default, unlike version |
cpp_domain_type_generator.cpp.mustache |
Recorded, not changed | Defaulting the int alone leaves issue_id uninitialized, like every uuid primary-key member. A template-level default for all primary-key members is a codegen decision. See the Notes entry. |
| 3 | --rows not a multiple of 10 silently truncates the family counts, leaving the last instrument block without its issue row |
load_trade_population.py main() |
Accepted | A guard after argparse exits with a clear message unless --rows is a multiple of 10, before any DB work. |
| 4 | counts[next(iter(family))] pulls an arbitrary set member to print the family count |
load_trade_population.py main() |
Accepted | A family_size computed once now feeds both the per-table counts and the header printout. |
| 5 | Import and save paths persist the slim instrument header only; the assembled issue and fact rows do not reach the DB | ore_import_execute_handler.cpp, OreImporter.cpp, ImportTradeDialog.cpp, BondInstrumentForm.cpp |
Recorded, not changed | Every site carries a comment naming the wave boundary. Issue find-or-create and the fact saves are the DB-bound import rework of unit 2; the full parts save is unit 3's qt bridge. |
| 6 | The forward mappers and the shell add mint one fresh issue per trade; nothing finds an existing issue by security_id |
bond_instrument_mapper.cpp, bond_instrument_commands.cpp |
Recorded, not changed | Repeated adds of one ISIN mint duplicate issue rows until the find-or-create DB boundary of unit 2 lands. Dedup by security_id exists only in the migration and the loader today. |
| 7 | The shell add writes the issue, instrument and fact rows through three saves with no transaction; a later failure leaves the earlier rows | bond_instrument_commands.cpp |
Recorded, not changed | The verb docstring and the failure messages name the rows that remain saved. The live-testing surface wants this visibility; wave 1.7 exercises the partial states. |
| 8 | The shell and qt surfaces lost the wide extension arguments (call_date, future_expiry_date, option_expiry_date, conversion_ratio, the ascot option type) |
bond_instrument_commands.*, BondInstrumentForm.* |
Recorded, not changed | The columns have no destination rows in this wave: future_expiry_date has no XSD-backed fact column (wave 1.1 survey), option_expiry_date feeds exercise-schedule child rows, call_date and conversion_ratio feed issue child rows by ISIN. The migration header lists the same set as not migrated. |
| 9 | Nothing cross-checks trade_type_code against the engaged fact row: a shell add can write a bond_option row under a Bond instrument |
bond_instrument_commands.cpp |
Recorded, not changed | The DB cannot see the mismatch: the fact tables carry no trade_type_code column. The verb trusts the operator to pair the code with the fact arguments; wave 1.7's per-code round trips decide whether a guard is needed. |
| 10 | A cleared extensions combo kept the staged fact, and both fact combos could engage at once | BondInstrumentForm.cpp writeUiToInstrument() |
Partially fixed | The empty-combo branches now reset data_.option and data_.trs to nullopt, so the container tracks the form. The both-set guard is declined: it is the code-vs-fact cross-check of row 9, and the save path persists the header row only this wave. |
| 11 | The shell TRS branch never assigned funding_leg_type, so every TRS add failed the NOT NULL and the two-value check on ores_trading_bond_trs_tbl |
bond_instrument_commands.cpp |
Fixed | The value derives from the funding leg code as the migration derives it: 'Fixed' is a fixed leg with no index, any other code is a floating leg carrying the code as the funding index. |
| 12 | reverse_bond_repo hardcoded a fixed, non-payer leg and discarded the captured repo_type, repo_rate and repo_index |
bond_instrument_mapper.cpp |
Fixed | The reverse now reads the repo fact. The same class in reverse_bond_trs (the captured fixed funding rate was dropped) is fixed in the same commit; both round-trip tests assert the captured values survive. |
| 13 | ImportTradeDialog stamped instrument_id and trade_id by hand instead of stamp_ids(), leaving the engaged fact rows unstamped |
ImportTradeDialog.cpp |
Fixed | The branch calls trading::domain::stamp_ids(), which stamps the fact rows too. |
| 14 | The header guard ORES_TRADING_DOMAIN_BOND_INSTRUMENT_DATA_HPP breaks the ORES_TRADING_API_DOMAIN_* convention of its directory |
bond_instrument_data.hpp |
Fixed | Renamed to ORES_TRADING_API_DOMAIN_BOND_INSTRUMENT_DATA_HPP. |
| 15 | The wave 1.6 draft claimed the aggregate "reaches the guidance only at 1000 rows", but four of the six samples are above the 2000 rows/sec guidance | task org, the wave 1.6 note and the Result | Fixed | Restated: the aggregate lands on both sides of the guidance, so the size alone does not predict which side a run falls on. |
| 16 | The draft read a 1.5% notify cost from the trade_types attribution, but disabling that trigger moved the reading the wrong way by 5% (and by 1.9% for bond_instruments) |
task org, the wave 1.6 note | Fixed | Restated as no measurable cost at that resolution, which agrees with the deliverable's 1.5%. |
| 17 | The draft called five legacy tables "the slowest tables in every run"; in the first 20000 run they are not the top five | task org, the wave 1.6 note | Fixed | Restated: the slowest five in three of the four runs, and inside the slow cluster in the fourth. |
| 18 | The wave 1.6 record is written while the task stays STARTED, so the Result carries a wave record rather than a closing statement | task org, the Result | Recorded, not changed | The plan asks for the loading-exercise numbers in the task Result. The Result is marked as the wave 1.6 record and the task closes with wave 1.7. |
| 19 | "The nine family tables carry 8864 to 10155 rows/sec" counts a table the quoted range does not span | task org, the front matter Now and the Result |
Fixed | "Family tables" in a loader rows/sec context means the eight child and fact tables, per wave 1.5's own sentence and their 2000-rows-each load. bond_instruments is reported apart at 7948 to 8423 rows/sec. Both occurrences now read "eight". |
| 20 | Table cells group thousands with commas while the prose around them does not | task org, the loading-exercise table | Recorded, not changed | The split is the doc's existing convention and predates this wave: tables group, prose does not, as in wave 1.5's "7123 rows/sec". |
| 21 | "nine entities over ten tables" counts a table the family does not have | task org, the Result; story org, the front matter Now |
Fixed | The family is nine tables: bond_issues, bond_instruments, the five fact tables and the two issue-keyed child tables, confirmed against the migration's own summary select and the live schema. ores_trading_ascots_tbl carries no bond prefix, which is what made the count read as ten. Both occurrences now read "nine". |
| 22 | The trade-type negative "fails at the instrument insert before any row lands" contradicts the partial-state finding and the rows on the schema | task org, the wave 1.7 note | Fixed | The negative writes its issue first and fails on the instrument save, which is the publisher. WT17-N01 is open with no instrument joining to it, one of the four issues the partial-state finding counts. |
| 23 | "one of the four the partial-state finding below counts" is missing a relative pronoun | task org, the wave 1.7 note | Fixed | Reads "one of the four that the partial-state finding below counts". |
9. Result
The task closes here with unit 1 complete. Every acceptance item at the top of this task is met, and the evidence for each is below or in the Notes.
Every family entity has an org and codegen emits its C++ and SQL:
the bond_issue parent, bond_instrument with the ten-code check
and the issue_id FK, the five per-product fact tables and the two
issue-keyed child tables, nine entities over nine tables, all on the
SCD2 spine. The old bond shape and its hand-written stack are
deleted. The tree builds and its tests pass at every merge point.
The migration script in projects/ores.sql/migration/ converts the
old single-table bond shape into the new rows and is validated
against an old-shape scratch copy: exit 0, instruments 1018, issues
1014, options 1, trs 2, ascots 101, call dates 102, orphans 0, with
the guards demoed on a pristine copy. The detail is in the Notes.
The population loader loads the family at the three standard sizes. It ran on a database recreated from HEAD (schema version 0.0.25, commit ba0eee6234), every run with zero findings and clean cleanup:
--rows |
Rows | Seconds | Aggregate rows/sec |
|---|---|---|---|
| 1000 | 41,800 | 6.80 | 6,147 |
| 1000 | 41,800 | 23.47 | 1,781 |
| 10000 | 418,000 | 127.09 | 3,289 |
| 10000 | 418,000 | 202.16 | 2,068 |
| 20000 | 836,000 | 589.10 | 1,419 |
| 20000 | 836,000 | 306.06 | 2,731 |
The eight family tables carry 8864 to 10155 rows/sec at 20000 rows
(2000 rows each), about four times the deliverable's 2200 rows/sec
baseline on the deployed schema and above its 2000 rows/sec sizing
guidance for a bulk path. bond_instruments carries 7948 to 8423
rows/sec in the full run and 8908 isolated.
The aggregate falls below the 2000 rows/sec guidance in two of the
six samples (1781 rows/sec at 1000 rows and 1419 at 20000). Neither
shortfall is a property of the new shape: five legacy tables run at
365 to 1098 rows/sec at 20000 rows, and trade_types pays a 7.8x
penalty in-run against its isolated 7530 rows/sec. That in-run
penalty is not yet isolated; the diagnosis belongs with the parent
analysis's performance work. Per the plan the split is the recorded
price of storing the truth, not a failure.
The family drift gate passes: all nine entities regenerate with zero diff.
The shell round trips pass for the family against the environment.
Eleven adds covered all ten trade type codes and their fact branches,
the paginated get and eleven history calls returned the right type
and issue id, and the eleven deletes soft-closed every row out of the
live view. The DB layer and the changed-event stream confirm each
step: 28 events on the prefixed subject over 14 instrument ids, one
save and one delete each. Four negatives on the CHECK gate each
tripped their own constraint. Three findings are recorded and
classified in the Notes: the shell delete leaves the fact row open,
funding_rate stores as NULL through the codegen sentinel, and the
add is not atomic. None of the three is fixed here; the first belongs
to the parent story's service-layer work and the other two are
recorded boundaries.
The per-table detail, the attribution evidence and the finding classifications are in the Notes.