Task: Cut the vector growth in the portfolio read
Table of Contents
This page documents a task in the Decide the XML strategy for ORE documents story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Reading a portfolio spends 32.70% of its instructions in the trade vector's realloc_append and 19.29% in the trade move constructor (callgrind, import_portfolio_all_ore_example_files_can_be_parsed, 4.03 billion instructions in total, binary built from PR 3565). The generated append_to in xml/generated/portfolio.cpp calls emplace_back with no size hint, so each regrowth moves every trade already read. Cut that cost.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Decide the XML strategy for ORE documents |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-19 |
3. Acceptance
- A callgrind run of the same test shows the vector growth share falling, with the before and after counts in the task Measurements.
- Every portfolio in the example corpus still round trips: the XML suites and the corpus tests pass.
- The fix lives in the generator or the engine, not in hand edits to generated files, so the drift check still passes.
4. Plan
(Implementation strategy. Written when work starts; key decisions
are distilled into the parent story's * Decisions at close, but the
plan itself stays — it is the historical record of what we did.)
5. Measurements
Every number this task claims about a tree, with the command that produced it
and the commit it was measured at. A number without both is not a measurement,
because a clean-up record describes a tree and trunk moves under a branch: a
consumer count, a gate's component count and a suite's size all change with no
code changing, and the reader of a bare number cannot tell which tree it came
from. Where the raw output matters, put it in an evidence/ file beside the
task and name that file in the Command column.
| What | Value | Command | Commit |
|---|---|---|---|
| sizeof of a trade | 346,624 bytes; the next widest record is 13,208 bytes | a throwaway program printing sizeof of every generated struct |
7515c50292 |
| Members of a trade, records held by value included | 11,537; the widest other record is 454 | generate_xml_bindings.py member count, printed once while choosing the threshold |
7515c50292 |
| Instructions, portfolio import test, before | 4,028,918,281; import_portfolio_with_context 2,910,831,513 |
valgrind --tool=callgrind ores.ore.core.tests "import_portfolio_all_ore_example_files_can_be_parsed" |
7515c50292 |
| Instructions, same test, after | 3,070,700,246; import_portfolio_with_context 1,952,347,014 |
same command | a6c7e6f7da |
| Instructions, same test, reserve only (throwaway) | 3,027,004,932 | same command, with a fixed reserve in append_to |
7515c50292 plus a hand edit |
| XML benchmark | scan, tape and pugi unchanged | ./benchmarks/xml/run.sh |
16f7dfda83 |
| Full test binary | 376 cases, 475,402 assertions, all pass | ores.ore.core.tests |
a6c7e6f7da |
6. Notes
Not decided, and the first step is to choose by measurement:
- Reserve a size. The binder does not know the child count ahead, so this needs a hint from the scanner or a heuristic reserve on the first append.
- Make the move cheap. The profile shows a large
memsetunder the append, so default construction of atrademay cost as much as the move. - Change the container for the repeated child, for example to a deque, which never moves an element. That changes the generated record type and every reader of it.
The same append_to template serves every repeated child, so a change there
touches all 13 tables. Prove the drift check and the corpus tests after it.
7. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
8. PRs
9. Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|
10. Result
The generator now counts the members of each record, records held by value
included. A record with 1,000 or more keeps its repeated children in a
std::deque, which never moves an element. Four portfolio records qualify:
trade, component_trade, derivative and component_sub_trade. The
widest record that stays a vector has 454 members, so the threshold sits in a
gap of more than 10,000. Only portfolio.cpp and portfolio.hpp change in
their tables. The other families gain a default template parameter and no other
change.
Reading the portfolio test falls from 4,028,918,281 to 3,070,700,246 instructions, 23.8%. The import function alone falls from 2,910,831,513 to 1,952,347,014, 32.9%, and the binder from 2,310,204,223 to 1,339,979,756, 42.0%. The 340 KB zero fill of each new trade stays, and a throwaway reserve shows no container could remove it: it reaches 3,027,004,932, within 1.4% of the deque.
What is left is the shape of trade. It holds one member for every instrument
kind, 346,624 bytes, and a trade uses one. The fill is 33% of the test. A fix
changes the record, not the container, so it is a separate task and not started.
A test pins the property: trades keep their address while the document grows.
Run against a plain std::vector, the same check fails. I confirmed that with a
separate program, not by building the old tree again.
The benchmark gate asked for a row because the generated headers sit under the watched path. The XML benchmark measures scanning and the tape, not the binder, so its numbers did not move.
The deque rule this task added is removed again by Store each trade kind in its own column: a trade no longer holds every kind, so no record is wide.