Task: Populate the commodity family

Table of Contents

This page documents a task in the Populate the ACME books with a realistic trade population — the remaining work story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

ACME's books hold commodity trades across all twenty-five routed types, which no other class can claim: every commodity type the catalogue declares routes to commodity_instrument. The family carries 5% of the population. It is the second planner after bond and it proves the family-neutral plan, so it lands early rather than by weight.

2. Status

Field Value
State DONE
Parent story Populate the ACME books with a realistic trade population — the remaining work
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-24

3. Acceptance

  • All twenty-five commodity trade types book a trade, and none is skipped.
  • The class share of the population matches its weight within tolerance.
  • The family exports as the ORE portfolio the engine reads, and reverse-maps without loss.
  • The report that prices the family runs against the forward curves and volatility surfaces it needs, or the task states which pricing it cannot prove yet.
  • Adding the planner changes no code outside the synthetic component.

4. Plan

(Implementation strategy. Written when work starts; key decisions are distilled into the parent story's * Decisions at close, but the plan itself stays — it is the historical record of what we did.)

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
The population cases with the commodity planner 20 cases passed, 423 assertions publish/bin/ores.synthetic.core.tests "[trade_population]" abe7cbff38
The whole synthetic core suite 66 cases passed, 1459 assertions, on a database rebuilt the same day publish/bin/ores.synthetic.core.tests abe7cbff38
A commodity population booked through the trading service 8 planned, 8 written, 0 already present synthetic generate-trades --config 11111111-2222-3333-4444-555555555555 a8c84243a6
A second run of the same population 8 planned, 0 written, 8 already present same a8c84243a6
The rows the run wrote 8 commodity instruments over six catalogue commodities and eight trade types, each with the unit its catalogue row states select commodity_code, unit, trade_type from ores_trading_commodity_instruments_tbl ... a8c84243a6
       

6. Notes

6.1. The importer maps eight of the twenty-five shapes

The catalogue routes all twenty-five commodity codes to commodity_instrument, but the ORE importer maps only eight shapes, at projects/ores.ore/core/src/domain/commodity_instrument_projection.cpp lines 50-53: CommodityForward, CommodityOption, CommoditySwap, CommoditySwaption, CommodityVarianceSwap, CommodityAveragePriceOption, CommodityOptionStrip and CommodityBasketOption.

The other seventeen route to the table and no projection reads or writes them: CommodityForwardVolatilityAgreement, IntradayPowerForward, CommodityDigitalOption, CommodityDigitalAveragePriceOption, CommodityAsianOption, CommoditySpreadOption, CommodityPairwiseVarianceSwap, CommodityBasketVarianceSwap, CommodityAccumulator, CommodityTaRF, CommodityWorstOfBasketSwap, CommodityBestEntryOption, CommodityWindowBarrierOption, CommodityGenericBarrierOption, CommodityRainbowOption, CommodityStrikeResettableOption and CommodityPosition.

So the planner books all twenty-five, because the table takes them, and the export-and-reverse-map proof covers the eight the importer maps. The seventeen are a gap in the importer and not in the planner.

6.2. The catalogue is a prerequisite

Nothing in refdata names a commodity. commodity_code is free text with no foreign key, no commodity lookup table exists, and every commodity reference table is empty on a fresh database. The task Seed a commodity catalogue supplies the vocabulary. Until it lands this planner has nothing to draw from, and inventing a commodity is the one thing the generator may not do.

6.3. The curves are not this task's

A commodity on no forward curve books, exports and then fails at pricing. The market data stories own the curves, so this task states which pricing it cannot prove instead of asserting a round trip it cannot show.

6.4. Why this task came back

It was folded into Make the trade population plan family-neutral on 2026-10-24, on the argument that the seam needs a second family to prove it. It returned on 2026-10-25: the seam task cannot close while holding a family that needs a catalogue it does not own, and a task that waits on another belongs in the backlog beside it, not inside a refactor.

6.5. Two gaps the proof exposed

Neither is this task's to close, and both are named so the next reader does not meet them cold.

The catalogue does not reach the tenant through a bundle. A fresh provision gives ACME an empty commodity catalogue, because refdata.commodities is not in a bundle the acme_demo profile opts into, and the planner correctly refuses. The proof published the dataset to the tenant directly with ores_refdata_publish_commodities_from_dq_fn. The real fix is a bundle membership, which belongs to whoever owns the profile.

The commodity population is not in the seed profile. The four configs the profile states are all bonds, so the proof cloned one into a commodity config with SQL, with the same book and the same 453 counterparties. ACME's profile should state a cross-asset population; until it does, a provision produces a bond book only.

6.6. The size column is `integer` and the C++ is `std::uint32_t`

A real defect found by running the provision, not by reading it. Recording a trade generation config fails with:

value "2167221318" is out of range for type integer

trades_per_book is declared :type: integer with :cpp_type: std::uint32_t in the config model. The SQL column is int4 and the C++ type is unsigned, so the value the writer sends does not fit and the insert is refused. The neighbouring integer entity fields use int, which is what the SQL is.

The fix is to declare int in the model, regenerate, and carry int through the scope, the planner and the seed. Until then every tenant provision stops at generate_trades, so no population of any asset class can be booked.

6.7. How the booking proof is run

Recorded so the steps survive an interruption. The environment is brought up in this order, and each step is a command rather than a note.

  1. compass services start. Twenty-three come up; ores.marketdata.service fails, and booking does not need it.
  2. compass shell -l projects/ores.shell/scripts/library/provisioning/how_do_i_provision_the_system_with_acme_corporation_holding_group.ores. The rebuild alone provisions no tenant: only system and a leftover synthetic_fomc test tenant exist before this runs.
  3. Publish refdata.commodities to the ACME tenant. The catalogue arrives per tenant through the Librarian bundles, so ACME needs its own copy or the planner refuses for an empty catalogue. If refdata.commodities is not in a bundle ACME opts into, it is published to that tenant directly for this proof, and the missing bundle membership is recorded as the gap.
  4. State a commodity generation config for ACME: a config row with asset_class_code = 'commodity', a book scope and a counterparty scope, cloned from one of the four bond configs the profile seeds.
  5. synthetic generate-trades --config <id> in the shell. The command exists in synthetic_commands.cpp and runs the population service, which is the only caller besides the tests.
  6. Prove the rows: ores_trading_commodity_instruments_tbl holds one row per booked trade, and ores_trading_trades_tbl counts them against the config.

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

8. PRs

PR Title
   

9. Review

Comment summary File Decision Notes
       

10. Result

A commodity population books through the trading service, which is what this task existed to prove.

Eight trades were planned and written in one run, and a second run wrote nothing and found all eight already present. The rows carry six of the catalogue's commodities — ICE:B, NYMEX:CL, NYMEX:CSX, NYMEX:NG, PM:XAUUSD and XLME:AH — across eight trade types spanning the importer's shapes, and each row states the unit its catalogue row does: barrels for the two crudes, MMBtu for gas, short tons for coal, tonnes for aluminium and ounces for gold.

The planner refuses when the catalogue holds nothing, and that refusal is what caught the missing bundle: the first run after provisioning failed with nothing can state a commodity, and that is the right answer to an empty catalogue rather than an invented commodity.

The task also found and fixed two defects of its own making: the population size was declared unsigned against an int4 column, which stopped every tenant provision at the config insert, and the generated commodity handlers were never composed, so the catalogue could not be read at all. Neither was visible to any test; both were caught by running the thing.

Not done: the commodity population is not in the ACME seed profile, and the catalogue is not in a bundle. Both are recorded above and belong to other owners.

11. Result

Emacs 29.3 (Org mode 9.6.15)