Rework an instrument family to the instrument table rule

1. Goal
2. Preconditions
3. The three principles
4. Steps
5. Pitfalls
6. Postconditions
7. See also

This page documents a runbook — a named, repeatable composition of recipes and skills for a complete multi-step procedure. Each step references a recipe or skill by id-link.

1. Goal

Take one instrument family from a wide table to a header, product fact tables and shared children, with exact decimals and an ORE round trip that comes back as the same rows.

2. Preconditions

The family has a story in the product backlog or the sprint, or you are about to promote one.
You have read section 14 of Investigation: the trade shape, data-oriented design and exact numbers and economic change.
The bond and rates families are done. Use them as the model: the bond pilot settled the shape, and the rates rework added the common-date view, the generated eventing tests and the exact digits.
You work one unit at a time. Each unit is a task, a branch and a PR. See Work a task through to merged PR.

3. The three principles

Apply these in every unit. They are the reason for most of the steps below.

Data-oriented approach. Organise the tables by how the code reads and writes the data, not by product taxonomy. Key every row by its trade. A list is rows of a child table. State the variation as a header with one fact table per product.
Split data by usage, hot and cold. Keep the fields read together on the main path in the header and the product fact table. Put rarely read data in its own table. A column that no mapper writes leaves the model. A column belongs on the header only if it holds for every code of the family.
Exact decimals for money. Every rate, spread, notional, amount, price, strike, fee and premium is a decimal in the domain type and numeric(38, 12) in the column. A double never enters the persisted domain. At the ORE boundary, keep the digits the document wrote. A continuous quantity (volatility, correlation, weight, probability, recovery rate, tranche fraction) stays a binary float, in the domain type and in the column; the two must agree.

4. Steps

4.1. Phase 0: set up the story

[ ] Promote the family's story from the backlog's next bucket into the sprint with git mv, so its identifier and history stay. Retag it, list it in the sprint's Stories table, and remove it from the backlog index.
[ ] Scaffold the first task, the analysis, with compass add task and start it.
[ ] Reset the task branch onto origin/main if compass task start switched to an old branch. Do this at the start of every unit.

4.2. Phase 1: analyse (no code)

The analysis is one task. It writes design.org and an ER diagram beside the story. The bond analysis found twelve findings and the rates analysis twenty; each is a finding with evidence.

[ ] Define the family. List the catalogue codes that route to the family. For each code, record whether an importer writes it and which corpus documents hold it. Count the documents in Example_Trades and in all example directories.
[ ] Inventory every surface. Models, SQL, domain C++, mappers, repositories, services, messaging, shell commands, history mapper, TypeScript protocol, the ORE mapper and the ORE binding structs, and the tests.
[ ] Make the use table. For each column of the current table, record which code writes it and which code reads it. This is the hot and cold split. Note the columns no mapper writes and the columns written and never read.
[ ] Compare with the ORE schema. For each product, list the elements the schema has and no column holds. Check that the export writes every required element. An export that omits a required element is not valid ORE.
[ ] Find the money. Classify every numeric column as money or continuous, as the exact-money check in the data-oriented design page does. List every money column and its type, and list the columns whose type and domain type disagree. List the ORE binding members that carry a money value and are bare floats. Each one is a digits task; a continuous member is not.
[ ] Write the findings. One row each, with the evidence that proves it. Check each number against the tree and put the command in the task's Measurements.
[ ] Propose the model. A header with the columns every code shares, one fact table per product keyed by the trade, shared children reused (legs, options, strikes, schedules), and a new child only where no shared shape fits. Record the rejected alternatives.
[ ] Draw the diagram. Write design_er.puml and render design_er.png. Read the picture.
[ ] Map the current model onto the target. Every current column gets a destination or a reason to drop it.
[ ] State what each surface must change and the common-date mapping.
[ ] Stage the rework into units that each end in a check. Scaffold each unit as a BACKLOG task.
[ ] Review the design with the data-model review skill, then decline or accept each finding in the task's Review table.

4.3. Phase 2: the units

The order below is the order that worked for rates, with the digits moved first because they change the goldens. Adjust it to the family, and keep each unit small enough to check.

[ ] Repair the routing. The catalogue's instrument column names the table each code routes to. A code with no importer has no table, so leave its instrument empty. Generate the routing from the catalogue; do not hand-write it.
[ ] Carry the digits. Name the family's money members (not its continuous quantities) in the --exact-decimal list of projects/ores.ore/scripts/regenerate_xml_bindings.sh, then regenerate the tables. Never edit a generated table by hand. Read each member with record_reader::decimal or own_decimal and write it back with set_decimal. Make every money column numeric(38, 12). Add a mapper test that holds the digits of each member, and regenerate the goldens that change.
[ ] Write the legs and children. Reuse the shared leg, option, strike and schedule tables. Add a nullable column to a shared table before you add a table that repeats it.
[ ] Author the header and the fact tables. One model each, with the family's codes in the header's check, the trade as the key, and the header's delete cascade naming every fact table and child. Pin each row to the anchor's trade type and party. Delete the old shape and every hand-written file it needed.
[ ] Close the round trip. Write the reverse mapping of every product from the stored rows. Write every required ORE element. Test each product's fields through the round trip.
[ ] Map the dates into the common-date view. Declare :common_date: on the header and on the fact tables keyed by the trade alone.
[ ] Generate the eventing tests. Set the eventing flag in the model's drawer. Set party_scoped_writes on the fact tables. Give a generated date pair a generator_expr that keeps them in order.
[ ] Prove the round trip over the corpus. Import each trade, export it alone, import it again, and compare the stored family rows. Pin the document and trade counts. Keep a closed set of known differences and fail it in both directions. Record the counts with their commands.

4.4. Phase 3: gates for every unit

Run these before the PR. Record the results in the PR's testing evidence.

[ ] Build the touched targets, then run their suites with .env loaded.
[ ] After a schema change, run compass db recreate -y -k. Do not write a migration.
[ ] Run both codegen drift gates on the committed tree: check_component_drift.py --all and check_component_drift.py --sweep. Do not use --component <name> alone. It regenerates the SQL, the C++ and the batch TypeScript for that component, but not ores.ts.domain, so a new entity's domain TypeScript is never rendered and the gate stays green while the twin is missing. That happened on this family's header, cliquet, outperformance and TaRF units, and another component's regeneration committed the files.
[ ] Run the codegen tests, the cmake sources check, check_handler_permissions.py and check_sql_hygiene.py.
[ ] Run projects/ores.codegen/validate_docs.sh and the misspell check.
[ ] Rebase onto origin/main before the PR. After a rebase, regenerate and run the drift check again, because the templates may have changed. Recreate the database if the schema moved.
[ ] Close the task bookkeeping before the PR: Result, Measurements, Review, story and sprint rows.
[ ] After the last unit, set the story DONE in both the story and the sprint.

5. Pitfalls

Each of these cost a round in the rates or credit work.

A stale task branch. compass task start may switch to a branch made from an old main. Reset it to origin/main first.
A float round trip that looks equal. Comparing the first import with the second hides a stored float artifact, because both carry it. Compare against the document's own digits.
A global writer change moves goldens. Regenerate the goldens by moving the old file aside and running the bootstrap, then read every changed line. Each must be numeric text.
A patch anchor that matches twice. The reapply script must refuse an anchor that is missing or not unique.
A generated test with no party scope. A fact table written under a trade needs party_scoped_writes, or its test aborts on an empty party.
Two random dates. A generator that draws a start and an end independently breaks a maturity > start check on some runs.
A pinned bug. A test that expects the float artifact keeps the bug alive. Correct it when you fix the member.
A claim without a command. A number in a design or a Result has its command and commit in Measurements, or it is not a measurement.
A design column the binding does not hold. The credit design gave the agreement a money column for its protection fee, and the binding showed a list of legs. Read the type of every ORE member before you copy a design column, and split a unit when the binding disagrees with the design.
A column with no writer. A fact column lands with the mapper that writes it. A column added early is cold data that nothing reads, and the next unit has to explain it.
A pinned gap fails when you fix it. A test that lists the documents the export cannot re-read fails when a unit fixes one. Remove the name from the list and from the backlog capture in the same commit.
A proof that passes first time. Give it a control: each trade must render a header row and a leg row, and a changed stored field must change the rendering.
A shared mapper change reaches other families. A cashflow leg changed the rates export too. Run the rates suites, and say so in the task.
A trailer in a quoted heredoc. A quoted heredoc does not expand a variable, so a commit trailer holds the text of the variable. Write the id out, or record it in a follow-up commit, because commits are never amended.
A stale database after a merge. A merge from main can add a column. When a suite fails on a missing column, recreate the database before you debug.

6. Postconditions

The family has a header, product fact tables and shared children, generated from models.
Every money value is stored and exported as the digits the document wrote.
The corpus proof passes with pinned counts and a closed set of known differences.
The old shape and its hand-written stack are gone, and no migration was written.
The story and the sprint row are DONE.

7. See also

Investigation: the trade shape, section 14, which states the instrument table rule.
Data-oriented design.
Exact numbers and economic change.
The component architecture audit, which a closed task walks over each component it touched.
Add a task to an existing story.