Story: Make NATS wire format configurable: JSON/MessagePack, decided once at startup
Table of Contents
- Goal
- Status
- Acceptance
- Tasks
- Decisions
- Architecture: non-polymorphic wire_codec, single startup decision, no per-message negotiation
- Config: ORES_NATS_WIRE_FORMAT, default msgpack, fail-fast on unrecognised value
- Server: process-wide default wire_codec, not per-call-site DI
- Blast radius was wider than the write-up's audit: service self-auth
- Shared primitive lives in ores.nats, not duplicated per client
- Embedded/persisted JSON string fields are exempt from wire_codec migration
- Out of scope
- Result
This page documents a story in Sprint 24. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
Goal
Every NATS message (server responses, service handler decode, Qt
client requests, shell client requests) is currently serialized by a
hard-coded, inline rfl::json::write=/=rfl::json::read call at each
individual send/receive site – roughly 40+ distinct call sites spread
across four codebases (ores.service, ores.qt, ores.shell,
ores.cli). This baked-in choice is why the image-batch story's
raw-bytes-not-base64 follow-up needed a bespoke, message-specific
binary framing: there was no general way to avoid JSON's base64
encoding of byte vectors anywhere in the stack.
This story decouples serialization format from the call sites
entirely. A single wire_codec value – concrete, non-polymorphic,
holding the chosen wire_format (json or msgpack, via
reflect-cpp's existing rfl::msgpack support) – is constructed
once, at process startup, from a .env-driven config value. It is
then threaded through every consuming layer via dependency injection;
no code anywhere inspects a message to decide its format at runtime,
and no per-message negotiation or auto-detection exists. Switching a
process's wire format becomes a one-line .env change plus a
restart, not a code change.
As a consequence, MessagePack's native binary type (which
reflect-cpp already maps std::vector<uint8_t> onto, with zero
custom code) eliminates the base64 tax for every message that carries
binary data, for every message type – not just images – once a
process's .env selects msgpack. This supersedes the narrower
raw-bytes-not-base64-for-images
task.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent sprint | Sprint 24 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-07-31 |
Acceptance
- A written design doc (this story's first task) captures the current blast-radius audit, the chosen architecture, and the rationale against alternatives (esp. per-message header-based negotiation, which this story deliberately does not do).
- A
wire_codectype inores.natswrapsrfl::json=/=rfl::msgpackbehind one interface, selected once at construction from a.env-drivenwire_formatvalue, with round-trip unit test coverage for both formats. ores.service::messaging::handler_helpers(server),ores.qt'sClientManager, andores.shell's command request helpers all route through the injectedwire_codecinstead of a hard-codedrfl::jsoncall.ores.shell's copy-pasteddo_request=/=do_auth_requesthelper (duplicated across ~13 command files) is consolidated into one shared helper as part of this migration.- End-to-end verification: a test environment configured for
msgpackround-trips correctly across Qt<->service and shell<->service, confirmed via the existing image-batch test scenarios (proving the raw-bytes-not-base64 win is realised for free).
Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Write up the wire-format architecture analysis and design decision | DONE | 2026-07-28 | 2026-07-28 | Document the current blast-radius audit (every NATS-relevant rfl::json call site across ores.service, ores.qt, ores.shell, plus confirming ores.cli is out of scope), the chosen architecture (a single non-polymorphic wire_codec constructed once at startup from .env, threaded via dependency injection, no per-message negotiation), and the rationale versus the rejected per-message header-negotiation alternative. This becomes the spec the later implementation tasks execute against. |
| Build wire_codec abstraction and startup config in ores.nats | DONE | 2026-07-29 | 2026-07-29 | Add a wire_format enum (json, msgpack), a .env-driven config value read once at startup, and a wire_codec class holding the chosen format at construction with template encode/decode methods dispatching to rfl::json or rfl::msgpack. Non-polymorphic, no per-message branching on message content – the format is fixed for the codec instance's lifetime. Round-trip unit tests for both formats. This is the shared foundation every other task in this story depends on. |
| Migrate ores.service messaging handler_helpers to wire_codec | DONE | 2026-07-29 | 2026-07-29 | Update reply() and decode() in ores.service/messaging/handler_helpers.hpp to route through an injected wire_codec instead of a hard-coded rfl::json call. This is the server-side choke point almost every service handler already goes through, so this single change flips every service's request/response handling in one pass. Audit for any service-to-service NATS calls that bypass these two helpers and migrate or document them separately. |
| Consolidate and migrate ores.qt ClientManager to wire_codec | DONE | 2026-07-29 | 2026-07-30 | ClientManager.hpp/.cpp plus ClientManagerExportPortfolio.cpp and ClientManagerTradeInstrument.cpp currently have around ten separate call sites hard-coding rfl::json (process_authenticated_request variants, send_authenticated_request variants, login, signup, event-cache decode, exportPortfolio, trade_instrument). Consolidate these into one or two shared private helpers first, reducing duplication as a side effect, then have those route through an injected wire_codec instead of rfl::json directly. |
| Consolidate and migrate ores.shell request helpers to wire_codec | DONE | 2026-07-30 | 2026-07-30 | The do_request/do_auth_request template helper is copy-pasted independently into thirteen shell command .cpp files (18 definitions, 60 call sites), plus 7 more files using other direct rfl::json patterns (101 occurrences across 22 files total). De-duplicate into one shared helper in a shell-wide header first – a real cleanup win independent of the format work – then have that single helper route through an injected wire_codec instead of rfl::json directly. |
| End-to-end verify msgpack wire format and close out the story | DONE | 2026-07-31 | 2026-07-31 | With every layer migrated, flip a test environment's .env to msgpack and verify round-trip correctness across Qt-to-service and shell-to-service, including the image-batch flows this whole investigation started from. Confirms the raw-bytes-not-base64-for-images task's goal is achieved for free. Abandon that task in favour of this story at close, and record the .env config knob in the relevant How do I recipe. |
| Migrate remaining service-to-service hard-coded rfl::json call sites to wire_codec | DONE | 2026-07-30 | 2026-07-30 | A repo-wide sweep (grep for rfl::json combined with request_sync usage) found service-to-service NATS call sites still hard-coding rfl::json instead of routing through ores::nats::default_wire_codec(), discovered outside every completed task's audited file list: ores.iam.core/service/cache/party_cache.hpp, ores.iam.core/messaging/tenant_handler.hpp, ores.marketdata.client/crm_client.cpp, ores.refdata.client/service/cache/currency_pair_convention_cache.hpp, ores.reporting.core/messaging/report_definition_template_handler.hpp, ores.trading.core/messaging/trade_handler.hpp, plus ClientManagerTradeInstrument.cpp's decode side (parse_trade_instrument, a JSON-specific two-phase parser in ores.qt.headless). Migrate every one onto the common wire_codec encode/decode helpers, then re-run the sweep to confirm it is clean, so a full-environment msgpack switch is safe. |
| Fix fx_spot_subscription hardcoded rfl::json, ignoring wire codec | DONE | 2026-08-02 | 2026-08-02 | fx_spot_subscription.cpp still hardcodes rfl::json::read regardless of ORES_NATS_WIRE_FORMAT, so FX Spot Monitor and Cross-Rates Matrix silently receive zero ticks under msgpack – missed by the story's migration sweep. |
Decisions
Architecture: non-polymorphic wire_codec, single startup decision, no per-message negotiation
Decided in the write-up task. A self-describing, per-message header
(mirroring the compression feature's X-Content-Encoding) was
considered and explicitly rejected in favour of a wire_codec value
type holding a wire_format fixed at construction from a .env
value, read once at process startup and threaded to every consumer
via dependency injection. Rationale: wire format is a binary,
environment-wide choice with no natural per-message trigger, unlike
compression's genuinely per-message, size-dependent decision – the
header approach's flexibility (safe mixed-fleet rollout) isn't a
requirement here, and .env is explicitly the intended control
surface. Full blast-radius audit (6 call sites in ores.service, ~10
plus stragglers in ores.qt, 101 across 22 files in ores.shell;
ores.cli confirmed out of scope) and the wire_codec type's shape
are recorded in that task's * Plan.
Config: ORES_NATS_WIRE_FORMAT, default msgpack, fail-fast on unrecognised value
Decided in the wire_codec build task. The --nats-wire-format
option (env ORES_NATS_WIRE_FORMAT, same per-service prefix
convention as the existing nats-tls-* options) defaulted to json
while every layer was being migrated, preserving pre-existing
behaviour for any process that didn't set it. With the
end-to-end verification task confirming every layer round-trips
correctly under msgpack (server-to-server, Qt client, and shell
client, including image-carrying flows), the default was flipped to
msgpack at this story's close – the whole point of making the
format configurable was to stop paying json's base64 tax on binary
payloads by default, not just to make it possible to opt in per
environment. An
unrecognised value throws std::invalid_argument at startup rather
than silently falling back – misconfiguration is a fail-fast error,
not a silent default. ores.nats.lib links reflectcpp::reflectcpp
publicly (previously only transitively/privately via
ores.utility.lib) since wire_codec.hpp is a public, header-only
template type every consumer needs rfl/json.hpp=/=rfl/msgpack.hpp
for.
Server: process-wide default wire_codec, not per-call-site DI
Decided in the handler_helpers migration task. decode<Req>(msg) is
called from ~250+ handler call sites across every service, none of
which have a wire_codec or client reference in scope – threading
one through would mean editing every call site, defeating this
task's "single change, no per-handler edits" goal. Resolved with
ores::nats::default_wire_codec()=/=set_default_wire_codec(): a
process-wide default set automatically by
nats::service::client's constructor (every server process
constructs exactly one client, from its resolved nats_options), so
handler_helpers.hpp's reply()=/=decode() (and the
heartbeat_publisher=/=workflow_helpers stragglers) read it with zero
edits anywhere else. This is a deliberate, narrow exception to
per-instance dependency injection – justified because wire format is
a single process-wide, decided-once-at-startup value, not a
per-instance or per-message choice, matching this story's core
architectural decision above.
Blast radius was wider than the write-up's audit: service self-auth
Discovered live during the ClientManager task's msgpack smoke test,
not by the write-up task's static audit:
ores.iam.client::service_token_provider – every service's
self-authentication call at startup – hard-coded rfl::json,
entirely outside any completed task's file scope. This is as
universal a blocker as a bug can be: a msgpack-configured process
couldn't authenticate itself, let alone serve a single request, until
fixed. Fixed in the same commit, same shape as every other fix in
this story (ores::nats::default_wire_codec()). Take-away for the
remaining tasks: the write-up's static grep-based audit, however
careful, could not find every service-to-service NATS call site –
prefer a live msgpack smoke test (see
the test scenario that caught this)
over trusting the audit alone before declaring the story done. A
repo-wide sweep at the same time found several more such call sites
(party_cache.hpp, tenant_handler.hpp, crm_client.cpp,
currency_pair_convention_cache.hpp,
report_definition_template_handler.hpp, trade_handler.hpp),
tracked as * Notes on the end-to-end-verify task rather than fixed
here, to keep this task's diff scoped to =ores.qt=/its one universal
blocker.
Shared primitive lives in ores.nats, not duplicated per client
Decided in the ores.shell migration task, on explicit request mid-task:
ores.qt::ClientManager and ores.shell each originally grew their
own thin encode/transport/decode wrapper around wire_codec
independently. Rather than leave two parallel copies, the actual
primitive (request_and_decode=/=authenticated_request_and_decode,
templated on Response=/=Request, returning rfl::Result<Response>)
moved down into ores.nats/service/request_helpers.hpp, callable by
any NATS consumer regardless of its error-reporting convention.
ores.shell's do_request=/=do_auth_request (std::optional plus an
ostream failure message) and ClientManager's unauthenticated
process_request()=/=testConnection()=/=signup() (std::expected)
both now call it directly; ClientManager's three header-scoped
authenticated overloads keep their own thinner
encode_request=/=decode_response wrappers, documented as
deliberate rather than accidental duplication, since
send_authenticated_request* must return raw bytes (not a decoded
value) to preserve the MSVC-C1202-avoiding TU isolation
ClientManagerExportPortfolio.cpp=/=ClientManagerTradeInstrument.cpp
already relied on.
Embedded/persisted JSON string fields are exempt from wire_codec migration
Decided in the migrate-remaining-call-sites task. A whole category of
rfl::json::write=/=read call sites are not wire-payload encoding at
all: workflow command_json=/=result_json (persisted DB columns,
published raw by workflow_engine::publish_command regardless of
wire_format), scheduler job_definition::action_payload, nested
request_json fields carrying an inner request as opaque text inside
an outer wire message, and the JWKS public-key bootstrap exchange
(deliberately JSON-only since it may run before a client's own
wire_codec is established). These are left as rfl::json by design,
not migrated – doing so would either be a category error (the field's
value happens to be JSON text, it isn't the transport encoding) or
require an unrelated DB schema change. Documented in
doc/knowledge/architecture/nats_wire_format.org as the standing
reference for this distinction, since it recurs on every future audit.
Also found and fixed one genuine producer/consumer mismatch along the
way: workflow_engine.cpp's on_step_completed=/=on_start_workflow
still hard-coded JSON decode while their publishers
(workflow_helpers.hpp, workflow_handler.cpp) already used
wire_codec – a latent bug under any msgpack configuration, now fixed.
Out of scope
ores.http– the public REST API must stay JSON for external client compatibility; handled separately if ever revisited.ores.wt– handled separately.- Per-message format negotiation, a content-type header, or any runtime auto-detection scheme – deliberately rejected in favour of a single startup-time decision (see the write-up/decision task for the full rationale versus that alternative).
- Database entity JSON columns and config-file parsing – unrelated
rfl::jsonusages, not NATS wire format. ores.cli– itsrfl::jsonusages appear to be config-file I/O, not NATS wire calls; the write-up task should confirm this and either drop it from scope entirely or file a small follow-up if any genuine NATS call sites are found there.
Result
All eight tasks landed: a single non-polymorphic wire_codec,
decided once at process startup from ORES_NATS_WIRE_FORMAT (default
msgpack) and threaded via dependency injection through
ores.service, ores.qt, and ores.shell, with the ores.shell
do_request=/=do_auth_request duplication consolidated along the way.
End-to-end msgpack verification passed across Qt<->service and
shell<->service, including image-carrying flows, confirming the
raw-bytes-not-base64 win landed for free. All tasks were already DONE
or ABANDONED; only the story's own State field had drifted and
needed flipping at sprint close.