Task: Cut the vector growth in the portfolio read

Table of Contents

This page documents a task in the Decide the XML strategy for ORE documents story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

Reading a portfolio spends 32.70% of its instructions in the trade vector's realloc_append and 19.29% in the trade move constructor (callgrind, import_portfolio_all_ore_example_files_can_be_parsed, 4.03 billion instructions in total, binary built from PR 3565). The generated append_to in xml/generated/portfolio.cpp calls emplace_back with no size hint, so each regrowth moves every trade already read. Cut that cost.

2. Status

Field Value
State DONE
Parent story Decide the XML strategy for ORE documents
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-19

3. Acceptance

  • A callgrind run of the same test shows the vector growth share falling, with the before and after counts in the task Measurements.
  • Every portfolio in the example corpus still round trips: the XML suites and the corpus tests pass.
  • The fix lives in the generator or the engine, not in hand edits to generated files, so the drift check still passes.

4. Plan

(Implementation strategy. Written when work starts; key decisions are distilled into the parent story's * Decisions at close, but the plan itself stays — it is the historical record of what we did.)

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
sizeof of a trade 346,624 bytes; the next widest record is 13,208 bytes a throwaway program printing sizeof of every generated struct 7515c50292
Members of a trade, records held by value included 11,537; the widest other record is 454 generate_xml_bindings.py member count, printed once while choosing the threshold 7515c50292
Instructions, portfolio import test, before 4,028,918,281; import_portfolio_with_context 2,910,831,513 valgrind --tool=callgrind ores.ore.core.tests "import_portfolio_all_ore_example_files_can_be_parsed" 7515c50292
Instructions, same test, after 3,070,700,246; import_portfolio_with_context 1,952,347,014 same command a6c7e6f7da
Instructions, same test, reserve only (throwaway) 3,027,004,932 same command, with a fixed reserve in append_to 7515c50292 plus a hand edit
XML benchmark scan, tape and pugi unchanged ./benchmarks/xml/run.sh 16f7dfda83
Full test binary 376 cases, 475,402 assertions, all pass ores.ore.core.tests a6c7e6f7da

6. Notes

Not decided, and the first step is to choose by measurement:

  • Reserve a size. The binder does not know the child count ahead, so this needs a hint from the scanner or a heuristic reserve on the first append.
  • Make the move cheap. The profile shows a large memset under the append, so default construction of a trade may cost as much as the move.
  • Change the container for the repeated child, for example to a deque, which never moves an element. That changes the generated record type and every reader of it.

The same append_to template serves every repeated child, so a change there touches all 13 tables. Prove the drift check and the corpus tests after it.

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

8. PRs

PR Title
#3579 [ore] Hold the repeated children of a wide record in a deque
#3576 [agile] Scaffold the portfolio vector growth task

9. Review

Comment summary File Decision Notes
       

10. Result

The generator now counts the members of each record, records held by value included. A record with 1,000 or more keeps its repeated children in a std::deque, which never moves an element. Four portfolio records qualify: trade, component_trade, derivative and component_sub_trade. The widest record that stays a vector has 454 members, so the threshold sits in a gap of more than 10,000. Only portfolio.cpp and portfolio.hpp change in their tables. The other families gain a default template parameter and no other change.

Reading the portfolio test falls from 4,028,918,281 to 3,070,700,246 instructions, 23.8%. The import function alone falls from 2,910,831,513 to 1,952,347,014, 32.9%, and the binder from 2,310,204,223 to 1,339,979,756, 42.0%. The 340 KB zero fill of each new trade stays, and a throwaway reserve shows no container could remove it: it reaches 3,027,004,932, within 1.4% of the deque.

What is left is the shape of trade. It holds one member for every instrument kind, 346,624 bytes, and a trade uses one. The fill is 33% of the test. A fix changes the record, not the container, so it is a separate task and not started.

A test pins the property: trades keep their address while the document grows. Run against a plain std::vector, the same check fails. I confirmed that with a separate program, not by building the old tree again.

The benchmark gate asked for a row because the generated headers sit under the watched path. The XML benchmark measures scanning and the tape, not the binder, so its numbers did not move.

The deque rule this task added is removed again by Store each trade kind in its own column: a trade no longer holds every kind, so no record is wide.

Emacs 29.3 (Org mode 9.6.15)