Story: Fix image-batch NATS payload overflow, add SVG compression

Table of Contents

This page documents a story in Sprint 24. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

Goal

Discovered while testing the vintage IR dataset story: country flags (and any other batch-loaded image set) were partially missing in the Qt client. Root-caused to two distinct issues:

  1. A now-fixed stale-binary bug: ores.assets.service queried a renamed column (svg_data -> mime_type=/=data) and had not been rebuilt since. (Fixed by rebuild; no code change needed for this part.)
  2. ImageCache's batch-fetch caps each NATS request at MAX_IMAGES_PER_REQUEST images by count, not byte size. SVG flags vary widely in size (a few KB for a simple flag, considerably more for a complex one, e.g. Norfolk Island or CEFTA), so a batch that happens to cluster several large images together can still exceed NATS's default 1MB max payload – and the entire batch silently fails when that happens, dropping every image in it. As an immediate mitigation, MAX_IMAGES_PER_REQUEST was lowered from 50 to 15 (this story's prerequisite commit) – reduces the odds without eliminating the pathological case.

Additionally: image data (SVG) is sent completely uncompressed – base64-encoded raw XML inside a JSON payload, no gzip/zstd anywhere in the pipeline. SVGs compress very well (typically 70-80% size reduction), so this is both a payload-size contributor and a bandwidth/latency cost independent of the batching bug.

This story's goal: make batching genuinely byte-size aware (track cumulative payload size while building each batch, not just count) so a batch is cut short whenever adding the next image would risk exceeding the NATS payload limit, and add compression to the image transfer pipeline (client and server) so payloads shrink regardless of batch size.

Alternative worth weighing: HTTP download instead of NATS request-reply

ores.compute.wrapper already downloads binaries via a plain HTTP GET against the generic S3-like object storage endpoint (ores.http.core/routes/storage_routes.hpp, GET /api/v1/storage/{bucket}/{key}) rather than a NATS request-reply round trip. Bulk binary transfer (many images, each independently addressable by key) is arguably a better fit for that same pattern than for NATS request-reply: no payload-size ceiling to chunk around at all, standard HTTP caching/range-request semantics apply for free, and it sidesteps the batching problem entirely rather than tuning around it.

Leaning towards staying on NATS unless the decision task turns up a good reason not to: NATS itself has no issue with binary payloads (a NATS message is just an opaque byte array), so there's no inherent transport ceiling being fought here. The actual cost stack today is self-inflicted by this app's own wire format, not by NATS or by using request-reply: every message (including image responses) goes through rfl::json::write=/=read (ores.service/messaging/handler_helpers.hpp), so binary SVG data is base64-encoded into a JSON string field – ~33% inflation – on top of being sent fully uncompressed. Sending raw (optionally gzip'd) bytes directly as the NATS payload for image- carrying messages, bypassing JSON/base64 for that field, plausibly removes most of the overflow risk on its own. The decision task should verify this against the synthetic worst case before treating the HTTP- transport migration as necessary – it's a materially bigger change (new storage-addressing scheme for images) that should only be taken on if NATS-with-compression genuinely can't be made to fit.

Status

Field Value
State DONE
Parent sprint Sprint 24
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-07-29

Acceptance

  • ImageCache's client-side batching tracks cumulative estimated payload size (accounting for base64 + JSON overhead, not just raw SVG bytes) and closes a batch early, before adding an image that would push it over a safe threshold below NATS's max payload.
  • The fix is verified against a synthetic worst case: many large SVGs (e.g. duplicate the largest known flag N times) requested in one fetch, confirming no batch ever exceeds the payload limit regardless of MAX_IMAGES_PER_REQUEST.
  • Image data is compressed on the wire (server compresses on write or send, client decompresses on receive) – gzip is sufficient, no need for a more exotic codec for text/SVG payloads this size.
  • MAX_IMAGES_PER_REQUEST can be raised back towards its original value (or removed as a meaningful constraint) once byte-size-aware batching and compression are both in place, since count alone is no longer the binding constraint.

Tasks

Task State Start End Description
Decide: NATS binary+compression vs. HTTP storage-endpoint transport for image delivery DONE 2026-07-26 2026-07-29 Confirm that sending images as raw (non-base64) compressed bytes over NATS resolves the payload-overflow problem on its own, before considering a move to the existing HTTP storage-endpoint transport. Record the decision and rationale.
Add gzip compression to the image NATS pipeline, send raw bytes instead of base64 DONE 2026-07-27 2026-07-27 Server compresses SVG image data before sending; client decompresses on receive. Stop routing binary image data through the JSON/base64 encoding used for the rest of the messaging protocol – send it as raw bytes on the wire for image-carrying messages, eliminating both the base64 inflation and the missing compression in one pass.
Make ImageCache batching byte-size aware DONE 2026-07-27 2026-07-28 Track cumulative estimated payload size (post-compression) while building each image-fetch batch, not just image count, so a batch is cut short before it would risk exceeding the NATS payload limit. Backstop for whatever pathological cases remain after compression; revisit MAX_IMAGES_PER_REQUEST once this and the compression task both land.
Make ImageCache batching byte-size aware, add SVG compression (ABANDONED – split above) ABANDONED     Superseded: bundled a design decision with two independent implementation concerns under one task.
Send raw bytes instead of base64 for image-carrying NATS messages (ABANDONED – superseded) ABANDONED 2026-07-28 2026-07-28 Superseded by the generic NATS wire format story: MessagePack solves the base64 problem for every message type, not just images, via a single .env-driven format choice.

Decisions

Stay on NATS request-reply; do not migrate to HTTP storage-endpoint transport

Decided in the decision task. Measured (not estimated) the actual worst-case batch – the 15 largest SVGs in external/flags/flag-icons/ (the real worst offenders turned out to be Serbia and Ascension Island, not the story's original examples of Norfolk Island/CEFTA, which are both small) – against NATS's actual 1 MiB default max payload (no override configured anywhere in this repo):

Encoding Bytes vs. 1 MiB limit
Raw, base64'd (today's wire format) 1,348,864 132% – overflows
Raw bytes, no base64, uncompressed 1,011,648 96% – no margin
Gzip, still base64'd 415,601 40%
Gzip, raw bytes (no base64) 311,701 30%

The worst case already overflows today, confirming the bug. Gzip compression alone (independent of the base64 question) fixes it with a ~2.5x margin; adding raw-byte encoding on top gets ~3.4x. Neither number comes close to justifying the bigger, riskier HTTP-transport migration (new storage-addressing scheme for images). Both remaining tasks proceed as scoped: compression + raw-byte wire encoding (the load-bearing fix), and byte-size-aware batching (defense-in-depth backstop, not load-bearing given the margin above).

Compression implemented generically at the NATS transport layer, not image-specifically

Decided in the compression task. Every NATS message (client and server, request-reply and pub/sub) already funnels through two choke points in ores.nats (make_msg() on send, extract_message() on receive). Added a size-thresholded (4096 bytes), header-flagged (X-Content-Encoding: gzip) compress/decompress pair there instead of in image-specific code – every message above the threshold benefits automatically, with zero changes to ImageCache, ClientManager, or any protocol/domain type, and the mechanism is unit-testable in isolation. This effectively delivers the "compress all NATS messages" capture below as infrastructure, without deciding to spend effort auditing every other message type's traffic patterns – it simply applies wherever a payload happens to be large enough.

Raw-bytes-instead-of-base64 (the compression task's other originally- named mechanism) was descoped as not load-bearing given the 2.5x margin gzip-with-base64 already provides; split into a separate follow-up task.

Out of scope

  • Auditing non-image NATS traffic to decide whether any of it should be deliberately restructured to benefit from compression (as opposed to benefiting automatically, for free, whenever it happens to cross the size threshold above) – separate, broader question, filed as a backlog capture: Consider compressing all NATS JSON messages, not just images.
  • Decompressing jetstream_admin peek payloads – unrelated to this story's own goal (it's an admin-tooling gap exposed as a side effect of the compression this story introduced, not something blocking the fix itself) – split into its own story at close: Fix jetstream_admin missing gzip decompression.

Emacs 29.3 (Org mode 9.6.15)