Task: Decide: NATS binary+compression vs. HTTP storage-endpoint transport for image delivery
Table of Contents
This page documents a task in the Fix image-batch NATS payload overflow, add SVG compression story. It captures the goal, current status, acceptance, and any notes or results.
Goal
Confirm, before either implementation task starts, that staying on NATS request-reply with raw (non-base64) compressed bytes for image payloads is sufficient to eliminate the overflow – rather than defaulting to it and discovering mid-implementation that it isn't.
Estimate the worst-case compressed payload size for a batch of the
largest known SVGs (e.g. Norfolk Island, CEFTA) after removing the
base64 layer and applying gzip, and compare against NATS's max
payload. If the estimate comfortably clears the limit (with the
byte-size-aware batching task as the remaining backstop), record that
as the decision and rationale in this task's * Plan, then distil it
into the parent story's * Decisions at close. Only escalate to the
HTTP storage-endpoint transport migration (a materially bigger change:
new storage-addressing scheme for images) if the estimate shows
compression alone isn't enough.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Fix image-batch NATS payload overflow, add SVG compression |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-07-29 |
Acceptance
[X]Worst-case payload estimated from real data, not just a rule of thumb.[X]Estimate compared against NATS's actual max payload.[X]Decision recorded below with rationale, ready to distil into the story's* Decisionsat close.
Plan
Data: NATS's actual max payload
No max_payload override exists anywhere in this repo's generated
NATS config (build/config/nats-*.conf) or in ores.compass's NATS
setup code, so the server default applies: 1,048,576 bytes (1 MiB).
Confirms the story's claim.
Data: worst-case SVG batch, measured (not estimated)
The story names Norfolk Island and CEFTA as "complex" examples, but
checking the actual source data (external/flags/flag-icons/, 276
SVGs, the source flags_generate_metadata_sql.py reads from) shows
those aren't the worst offenders – nf.svg is 5.6 KB, cefta.svg is
910 B. The real top of the distribution is far larger:
| Flag | Raw bytes |
|---|---|
| rs.svg (Serbia) | 181,634 |
| sh-ac.svg (Ascension) | 143,373 |
| bo.svg (Bolivia) | 102,880 |
| mx.svg (Mexico) | 84,753 |
| es.svg (Spain) | 80,958 |
| (10 more, descending) |
Measured the actual worst case a batch of MAX_IMAGES_PER_REQUEST
(currently 15) could construct – the 15 largest SVGs in the corpus,
gzip -9'd for real rather than assuming the "70-80%" rule of thumb:
| Encoding | Bytes | vs. 1 MiB limit |
|---|---|---|
| Raw, base64'd (today's wire format) | 1,348,864 | 132% – overflows |
| Raw bytes, no base64, uncompressed | 1,011,648 | 96% – still tight |
| Gzip -9, still base64'd | 415,601 | 40% |
| Gzip -9, raw bytes (no base64) | 311,701 | 30% |
Finding
The worst-case 15-image batch already exceeds NATS's 1 MiB limit today (1,348,864 bytes, ~1.29x over) purely from base64 inflation on top of raw SVG size – confirming the bug is real and not just a theoretical edge case, and that even removing base64 alone (without compression) leaves it at 96% of the limit with no margin.
Adding gzip compression – even without touching the base64 layer – drops the same worst case to ~406 KB (40% of the limit, a comfortable ~2.5x margin). Doing both (raw bytes + gzip) drops it further to ~305 KB (30% of the limit, ~3.4x margin). Either way, compression alone resolves the overflow with room to spare; base64 removal is worth doing regardless (it is free bandwidth/CPU savings and the "decode what you just encoded for no reason" cost the story flagged), but is not itself load-bearing for fixing the bug.
Decision
Stay on NATS request-reply. Implement both remaining tasks as scoped:
- Compression + raw-byte wire encoding for image-carrying messages (this is what actually fixes the overflow, with a ~2.5-3.4x margin against the worst measured case).
- Byte-size-aware batching as the defense-in-depth backstop — not load-bearing given the margin above, but cheap insurance against a future even-larger SVG landing in the corpus, or a batch size larger than 15 being reintroduced later.
The HTTP storage-endpoint transport migration is not warranted: it would be a materially bigger change (new storage-addressing scheme for images, touching the client's fetch path end to end) for a problem that compression alone already solves with a comfortable margin. Revisit only if a future single SVG (not a batch) somehow approaches 1 MiB on its own, which no image in the current corpus is remotely close to (the single largest, rs.svg at 181,634 bytes raw, is 49,978 bytes gzip'd – under 5% of the limit on its own).
Notes
Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario, run
through the QA Validation Runner panel) that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
PRs
| PR | Title |
|---|---|
| #1733 | [agile] Close image-batch payload story, split off jetstream_admin decompress |
Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | Stale "no PR raised" sentence in this task's Result, contradicted by the #+pr/PR-table fields | task_decide-nats-compression-vs-http-transport.org | Fixed | Updated Result to reflect PR #1733. |
| 2 | #+updated/Last touched not bumped despite this PR's changes to the file | task_decide-nats-compression-vs-http-transport.org | Fixed | Bumped both to 2026-07-29. |
| 3 | Split story's "Next" status field reads "Break the story into tasks" though one task already carried over | fix-jetstream-admin-decompress/story.org | Fixed | Changed to "Start the decompress task." |
Result
Decision: stay on NATS request-reply; do not migrate image delivery
to the HTTP storage-endpoint transport. Measured the actual worst-case
batch (the 15 largest SVGs in external/flags/flag-icons/, not the
story's anecdotal examples, which turned out not to be the worst
offenders) and found it already overflows NATS's 1 MiB default limit
today under the current base64+uncompressed wire format
(1,348,864 bytes, ~132% of the limit) – confirming the bug is real.
Gzip compression alone (independent of the base64 question) drops the
same worst case to ~406 KB (40% of the limit, ~2.5x margin); doing
both (raw bytes + gzip) drops it to ~312 KB (30%, ~3.4x margin). All
acceptance criteria met; full data and rationale in * Plan above.
PR #1733 raised for this task alongside the story closure and the jetstream_admin follow-up split – the two implementation tasks it unblocked had already landed separately beforehand.