Task: Profile the portfolio read again and measure peak memory
Table of Contents
This page documents a task in the Decide the XML strategy for ORE documents story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Profile the portfolio read test, remove the largest avoidable cost, and state the peak memory of a large import.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Decide the XML strategy for ORE documents |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-20 |
3. Acceptance
- A callgrind profile names the costs after the trade columns
- The largest avoidable cost is removed and measured
- Peak memory of a large import is measured and recorded
4. Plan
Run callgrind on import_portfolio_all_ore_example_files_can_be_parsed from the
repository root, because the test finds the example files by a relative path.
Name the costs. Remove the largest cost that is avoidable. Run massif to measure
the peak heap.
(Implementation strategy. Written when work starts; key decisions
are distilled into the parent story's * Decisions at close, but the
plan itself stays — it is the historical record of what we did.)
5. Measurements
Every number this task claims about a tree, with the command that produced it
and the commit it was measured at. A number without both is not a measurement,
because a clean-up record describes a tree and trunk moves under a branch: a
consumer count, a gate's component count and a suite's size all change with no
code changing, and the reader of a bare number cannot tell which tree it came
from. Where the raw output matters, put it in an evidence/ file beside the
task and name that file in the Command column.
| What | Value | Command | Commit |
|---|---|---|---|
| Instructions of the test before the fix | 1,864,312,400 | valgrind --tool=callgrind ores.ore.core.tests import_portfolio_all_ore_example_files_can_be_parsed |
f4f57e27ce |
| Instructions of the test after the fix | 1,646,496,411 (-11.7 percent) | same | 0f8d099aac |
Cost of string::_M_construct<istreambuf_iterator> before the fix |
217,513,662 (11.7 percent) | callgrind_annotate |
f4f57e27ce |
Cost of file::read_content after the fix, 374 calls |
7,226,633 | callgrind_annotate --tree=calling --inclusive=yes |
0f8d099aac |
| SCRAM login of the test database fixture | 835,584,739 (50.7 percent of the run) | callgrind_annotate --inclusive=yes, PQconnectdb |
0f8d099aac |
import_portfolio_with_context, 374 calls |
528,135,079 (32.1 percent) | same | 0f8d099aac |
of which portfolio_projection::read (the bind) |
295,443,333 | same | 0f8d099aac |
of which portfolio_projection::map |
209,879,245 | same | 0f8d099aac |
of which read_rate_instrument, 2,668 calls |
157,286,395 | same | 0f8d099aac |
of which append_to<leg_data>, 3,624 calls |
48,046,747 | same | 0f8d099aac |
Peak heap of a small import (import_portfolio_from_minimal_swap) |
25,985,459 bytes | valgrind --tool=massif, largest mem_heap_B |
0f8d099aac |
| Peak heap of the all-examples import | 40,539,918 bytes | same | 0f8d099aac |
| Largest portfolio file | 621,090 bytes, 314 trades (ois_portfolio.xml) |
grep -c "<Trade " |
0f8d099aac |
Size of trade_kind_columns |
3,576 bytes | task "Store each trade kind in its own column" | ec1c3ca051 |
6. Notes
- The 466 million instructions of
sha256_block_data_order_avx2are not XML work. The test fixture opens database connections, and each login runs the SCRAM password exchange. The fixture costs 835 million instructions in all. That is more than the import it serves. - The fix is in
file::read_content, which the importer calls. A seekable stream now reads in one bulk read. A stream that cannot seek keeps the iterator copy. Two new tests cover a stream that starts mid-way and a stream that cannot seek. - Each holder of a trade owns one
trade_kind_columnsof 3,576 bytes. A portfolio document has one holder for all its trades, so the cost is 3.5 KB per document and not per trade. A composite trade adds one holder per component that holds a trade. - Before the columns, a trade was 346,624 bytes. The 314 trades of the largest file would have needed 108.8 MB for the trade vector alone. This is arithmetic, not a measurement. The measured peak for the whole import is 14.5 MB above the small-import baseline.
- Candidate:
read_rate_instrumentcosts about 59,000 instructions per trade. It is the largest part of the map. A profile of that function by line would show whether it is string work or decimal work. - Candidate:
append_to<leg_data>costs about 13,000 instructions per leg. Theleg_datarecord is large, and each append zero-fills it.
7. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
8. PRs
| PR | Title |
|---|---|
| #3590 | [platform] Read a seekable stream in one bulk read |
9. Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|
10. Result
The file read copied each file one character at a time and cost 11.7 percent of the portfolio read test. A seekable stream now reads in one call. The test falls from 1,864,312,400 to 1,646,496,411 instructions. Peak heap of the all-examples import is 40.5 MB, 14.5 MB above a small import. The trade columns cost 3.5 KB per document, not per trade. The affected test binaries pass: platform, marketdata core, testing, storage core and ore core.