Story: Make NATS wire format configurable: JSON/MessagePack, decided once at startup

Table of Contents

This page documents a story in Sprint 24. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

Goal

Every NATS message (server responses, service handler decode, Qt client requests, shell client requests) is currently serialized by a hard-coded, inline rfl::json::write=/=rfl::json::read call at each individual send/receive site – roughly 40+ distinct call sites spread across four codebases (ores.service, ores.qt, ores.shell, ores.cli). This baked-in choice is why the image-batch story's raw-bytes-not-base64 follow-up needed a bespoke, message-specific binary framing: there was no general way to avoid JSON's base64 encoding of byte vectors anywhere in the stack.

This story decouples serialization format from the call sites entirely. A single wire_codec value – concrete, non-polymorphic, holding the chosen wire_format (json or msgpack, via reflect-cpp's existing rfl::msgpack support) – is constructed once, at process startup, from a .env-driven config value. It is then threaded through every consuming layer via dependency injection; no code anywhere inspects a message to decide its format at runtime, and no per-message negotiation or auto-detection exists. Switching a process's wire format becomes a one-line .env change plus a restart, not a code change.

As a consequence, MessagePack's native binary type (which reflect-cpp already maps std::vector<uint8_t> onto, with zero custom code) eliminates the base64 tax for every message that carries binary data, for every message type – not just images – once a process's .env selects msgpack. This supersedes the narrower raw-bytes-not-base64-for-images task.

Status

Field Value
State DONE
Parent sprint Sprint 24
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-07-31

Acceptance

  • A written design doc (this story's first task) captures the current blast-radius audit, the chosen architecture, and the rationale against alternatives (esp. per-message header-based negotiation, which this story deliberately does not do).
  • A wire_codec type in ores.nats wraps rfl::json=/=rfl::msgpack behind one interface, selected once at construction from a .env-driven wire_format value, with round-trip unit test coverage for both formats.
  • ores.service::messaging::handler_helpers (server), ores.qt's ClientManager, and ores.shell's command request helpers all route through the injected wire_codec instead of a hard-coded rfl::json call.
  • ores.shell's copy-pasted do_request=/=do_auth_request helper (duplicated across ~13 command files) is consolidated into one shared helper as part of this migration.
  • End-to-end verification: a test environment configured for msgpack round-trips correctly across Qt<->service and shell<->service, confirmed via the existing image-batch test scenarios (proving the raw-bytes-not-base64 win is realised for free).

Tasks

Task State Start End Description
Write up the wire-format architecture analysis and design decision DONE 2026-07-28 2026-07-28 Document the current blast-radius audit (every NATS-relevant rfl::json call site across ores.service, ores.qt, ores.shell, plus confirming ores.cli is out of scope), the chosen architecture (a single non-polymorphic wire_codec constructed once at startup from .env, threaded via dependency injection, no per-message negotiation), and the rationale versus the rejected per-message header-negotiation alternative. This becomes the spec the later implementation tasks execute against.
Build wire_codec abstraction and startup config in ores.nats DONE 2026-07-29 2026-07-29 Add a wire_format enum (json, msgpack), a .env-driven config value read once at startup, and a wire_codec class holding the chosen format at construction with template encode/decode methods dispatching to rfl::json or rfl::msgpack. Non-polymorphic, no per-message branching on message content – the format is fixed for the codec instance's lifetime. Round-trip unit tests for both formats. This is the shared foundation every other task in this story depends on.
Migrate ores.service messaging handler_helpers to wire_codec DONE 2026-07-29 2026-07-29 Update reply() and decode() in ores.service/messaging/handler_helpers.hpp to route through an injected wire_codec instead of a hard-coded rfl::json call. This is the server-side choke point almost every service handler already goes through, so this single change flips every service's request/response handling in one pass. Audit for any service-to-service NATS calls that bypass these two helpers and migrate or document them separately.
Consolidate and migrate ores.qt ClientManager to wire_codec DONE 2026-07-29 2026-07-30 ClientManager.hpp/.cpp plus ClientManagerExportPortfolio.cpp and ClientManagerTradeInstrument.cpp currently have around ten separate call sites hard-coding rfl::json (process_authenticated_request variants, send_authenticated_request variants, login, signup, event-cache decode, exportPortfolio, trade_instrument). Consolidate these into one or two shared private helpers first, reducing duplication as a side effect, then have those route through an injected wire_codec instead of rfl::json directly.
Consolidate and migrate ores.shell request helpers to wire_codec DONE 2026-07-30 2026-07-30 The do_request/do_auth_request template helper is copy-pasted independently into thirteen shell command .cpp files (18 definitions, 60 call sites), plus 7 more files using other direct rfl::json patterns (101 occurrences across 22 files total). De-duplicate into one shared helper in a shell-wide header first – a real cleanup win independent of the format work – then have that single helper route through an injected wire_codec instead of rfl::json directly.
End-to-end verify msgpack wire format and close out the story DONE 2026-07-31 2026-07-31 With every layer migrated, flip a test environment's .env to msgpack and verify round-trip correctness across Qt-to-service and shell-to-service, including the image-batch flows this whole investigation started from. Confirms the raw-bytes-not-base64-for-images task's goal is achieved for free. Abandon that task in favour of this story at close, and record the .env config knob in the relevant How do I recipe.
Migrate remaining service-to-service hard-coded rfl::json call sites to wire_codec DONE 2026-07-30 2026-07-30 A repo-wide sweep (grep for rfl::json combined with request_sync usage) found service-to-service NATS call sites still hard-coding rfl::json instead of routing through ores::nats::default_wire_codec(), discovered outside every completed task's audited file list: ores.iam.core/service/cache/party_cache.hpp, ores.iam.core/messaging/tenant_handler.hpp, ores.marketdata.client/crm_client.cpp, ores.refdata.client/service/cache/currency_pair_convention_cache.hpp, ores.reporting.core/messaging/report_definition_template_handler.hpp, ores.trading.core/messaging/trade_handler.hpp, plus ClientManagerTradeInstrument.cpp's decode side (parse_trade_instrument, a JSON-specific two-phase parser in ores.qt.headless). Migrate every one onto the common wire_codec encode/decode helpers, then re-run the sweep to confirm it is clean, so a full-environment msgpack switch is safe.
Fix fx_spot_subscription hardcoded rfl::json, ignoring wire codec DONE 2026-08-02 2026-08-02 fx_spot_subscription.cpp still hardcodes rfl::json::read regardless of ORES_NATS_WIRE_FORMAT, so FX Spot Monitor and Cross-Rates Matrix silently receive zero ticks under msgpack – missed by the story's migration sweep.

Decisions

Architecture: non-polymorphic wire_codec, single startup decision, no per-message negotiation

Decided in the write-up task. A self-describing, per-message header (mirroring the compression feature's X-Content-Encoding) was considered and explicitly rejected in favour of a wire_codec value type holding a wire_format fixed at construction from a .env value, read once at process startup and threaded to every consumer via dependency injection. Rationale: wire format is a binary, environment-wide choice with no natural per-message trigger, unlike compression's genuinely per-message, size-dependent decision – the header approach's flexibility (safe mixed-fleet rollout) isn't a requirement here, and .env is explicitly the intended control surface. Full blast-radius audit (6 call sites in ores.service, ~10 plus stragglers in ores.qt, 101 across 22 files in ores.shell; ores.cli confirmed out of scope) and the wire_codec type's shape are recorded in that task's * Plan.

Config: ORES_NATS_WIRE_FORMAT, default msgpack, fail-fast on unrecognised value

Decided in the wire_codec build task. The --nats-wire-format option (env ORES_NATS_WIRE_FORMAT, same per-service prefix convention as the existing nats-tls-* options) defaulted to json while every layer was being migrated, preserving pre-existing behaviour for any process that didn't set it. With the end-to-end verification task confirming every layer round-trips correctly under msgpack (server-to-server, Qt client, and shell client, including image-carrying flows), the default was flipped to msgpack at this story's close – the whole point of making the format configurable was to stop paying json's base64 tax on binary payloads by default, not just to make it possible to opt in per environment. An unrecognised value throws std::invalid_argument at startup rather than silently falling back – misconfiguration is a fail-fast error, not a silent default. ores.nats.lib links reflectcpp::reflectcpp publicly (previously only transitively/privately via ores.utility.lib) since wire_codec.hpp is a public, header-only template type every consumer needs rfl/json.hpp=/=rfl/msgpack.hpp for.

Server: process-wide default wire_codec, not per-call-site DI

Decided in the handler_helpers migration task. decode<Req>(msg) is called from ~250+ handler call sites across every service, none of which have a wire_codec or client reference in scope – threading one through would mean editing every call site, defeating this task's "single change, no per-handler edits" goal. Resolved with ores::nats::default_wire_codec()=/=set_default_wire_codec(): a process-wide default set automatically by nats::service::client's constructor (every server process constructs exactly one client, from its resolved nats_options), so handler_helpers.hpp's reply()=/=decode() (and the heartbeat_publisher=/=workflow_helpers stragglers) read it with zero edits anywhere else. This is a deliberate, narrow exception to per-instance dependency injection – justified because wire format is a single process-wide, decided-once-at-startup value, not a per-instance or per-message choice, matching this story's core architectural decision above.

Blast radius was wider than the write-up's audit: service self-auth

Discovered live during the ClientManager task's msgpack smoke test, not by the write-up task's static audit: ores.iam.client::service_token_provider – every service's self-authentication call at startup – hard-coded rfl::json, entirely outside any completed task's file scope. This is as universal a blocker as a bug can be: a msgpack-configured process couldn't authenticate itself, let alone serve a single request, until fixed. Fixed in the same commit, same shape as every other fix in this story (ores::nats::default_wire_codec()). Take-away for the remaining tasks: the write-up's static grep-based audit, however careful, could not find every service-to-service NATS call site – prefer a live msgpack smoke test (see the test scenario that caught this) over trusting the audit alone before declaring the story done. A repo-wide sweep at the same time found several more such call sites (party_cache.hpp, tenant_handler.hpp, crm_client.cpp, currency_pair_convention_cache.hpp, report_definition_template_handler.hpp, trade_handler.hpp), tracked as * Notes on the end-to-end-verify task rather than fixed here, to keep this task's diff scoped to =ores.qt=/its one universal blocker.

Shared primitive lives in ores.nats, not duplicated per client

Decided in the ores.shell migration task, on explicit request mid-task: ores.qt::ClientManager and ores.shell each originally grew their own thin encode/transport/decode wrapper around wire_codec independently. Rather than leave two parallel copies, the actual primitive (request_and_decode=/=authenticated_request_and_decode, templated on Response=/=Request, returning rfl::Result<Response>) moved down into ores.nats/service/request_helpers.hpp, callable by any NATS consumer regardless of its error-reporting convention. ores.shell's do_request=/=do_auth_request (std::optional plus an ostream failure message) and ClientManager's unauthenticated process_request()=/=testConnection()=/=signup() (std::expected) both now call it directly; ClientManager's three header-scoped authenticated overloads keep their own thinner encode_request=/=decode_response wrappers, documented as deliberate rather than accidental duplication, since send_authenticated_request* must return raw bytes (not a decoded value) to preserve the MSVC-C1202-avoiding TU isolation ClientManagerExportPortfolio.cpp=/=ClientManagerTradeInstrument.cpp already relied on.

Embedded/persisted JSON string fields are exempt from wire_codec migration

Decided in the migrate-remaining-call-sites task. A whole category of rfl::json::write=/=read call sites are not wire-payload encoding at all: workflow command_json=/=result_json (persisted DB columns, published raw by workflow_engine::publish_command regardless of wire_format), scheduler job_definition::action_payload, nested request_json fields carrying an inner request as opaque text inside an outer wire message, and the JWKS public-key bootstrap exchange (deliberately JSON-only since it may run before a client's own wire_codec is established). These are left as rfl::json by design, not migrated – doing so would either be a category error (the field's value happens to be JSON text, it isn't the transport encoding) or require an unrelated DB schema change. Documented in doc/knowledge/architecture/nats_wire_format.org as the standing reference for this distinction, since it recurs on every future audit. Also found and fixed one genuine producer/consumer mismatch along the way: workflow_engine.cpp's on_step_completed=/=on_start_workflow still hard-coded JSON decode while their publishers (workflow_helpers.hpp, workflow_handler.cpp) already used wire_codec – a latent bug under any msgpack configuration, now fixed.

Out of scope

  • ores.http – the public REST API must stay JSON for external client compatibility; handled separately if ever revisited.
  • ores.wt – handled separately.
  • Per-message format negotiation, a content-type header, or any runtime auto-detection scheme – deliberately rejected in favour of a single startup-time decision (see the write-up/decision task for the full rationale versus that alternative).
  • Database entity JSON columns and config-file parsing – unrelated rfl::json usages, not NATS wire format.
  • ores.cli – its rfl::json usages appear to be config-file I/O, not NATS wire calls; the write-up task should confirm this and either drop it from scope entirely or file a small follow-up if any genuine NATS call sites are found there.

Result

All eight tasks landed: a single non-polymorphic wire_codec, decided once at process startup from ORES_NATS_WIRE_FORMAT (default msgpack) and threaded via dependency injection through ores.service, ores.qt, and ores.shell, with the ores.shell do_request=/=do_auth_request duplication consolidated along the way. End-to-end msgpack verification passed across Qt<->service and shell<->service, including image-carrying flows, confirming the raw-bytes-not-base64 win landed for free. All tasks were already DONE or ABANDONED; only the story's own State field had drifted and needed flipping at sprint close.

Emacs 29.3 (Org mode 9.6.15)