Task: Implement ores.shell market data import command
This page documents a task in the ores.shell command for market data import story. It captures the goal, current status, acceptance, and any notes or results.
Goal
Add a command to ores.shell that reads a market-data file (ORE
market.txt=/=fixings.txt format) and an optional --source tag, then
sends import_market_data_request over NATS
(marketdata.v1.import) exactly as ImportTradeDialog does today —
same request/response, same import_service, just a non-interactive
entry point.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | ores.shell command for market data import |
| Now | Done. |
| Waiting on | Nothing. |
| Next | Resume Import the 2016-02-05 ORE reference vintage; blocked on Full market.txt import hits duplicate observation key first. |
| Last touched | 2026-07-03 |
Acceptance
ores.shellgains an import command taking--file=/–fixings= paths and an optional--sourcetag. ✓- Round-trips successfully against a representative ORE example
vintage. ✓ — verified against
Academy/TA002_IR_SwapandMarketRisk/HistSimVar(including real fixings import); the originally-namedLegacy/Example_56vintage still fails, but on a separate, pre-existingimport_servicebug (Full market.txt import hits duplicate observation key) unrelated to this command — the command itself sends the request correctly in all cases. - A recipe documents the command, following the existing
ores.shellcommand doc convention. ✓ — three recipes, one per tested vintage.
Plan
(Implementation strategy. Written when work starts; key decisions
are distilled into the parent story's * Decisions at close, but the
plan itself stays — it is the historical record of what we did.)
No marketdata_commands.* exists yet in ores.shell — this is a new
command file, modelled closely on
projects/ores.shell/src/app/commands/bundles_commands.{hpp,cpp}
(its process_publish is the template: parse_args, an
authenticated_request + rfl::json round-trip via a local
do_auth_request<Response> helper, success/failure output). No
existing shell command currently slurps a file into a std::string
for a NATS payload (the only file reader, script_commands.cpp's
load, streams line-by-line instead), so a small local file-read
helper is new code here, not reuse.
- New header
projects/ores.shell/include/ores.shell/app/commands/marketdata_commands.hpp- Class
marketdata_commandswithlogger_name,register_commands(cli::Menu&, nats_client&)andprocess_import(std::ostream&, nats_client&, const std::vector<std::string>&), mirroringbundles_commands.hpp's shape.
- Class
- New source
projects/ores.shell/src/app/commands/marketdata_commands.cpp- Includes
ores.marketdata.api/messaging/import_protocol.hpp,ores.nats/domain/message.hpp,command_args.hpp,command_feedback.hpp,rfl/json.hpp. - Local anonymous-namespace helpers: a
do_auth_request<Response>copy (same pattern as bundles_commands.cpp — no shared header exports it) and aread_file(const std::string& path) -> std::optional<std::string>that opens anstd::ifstream, returnsstd::nullopton failure, else slurps viastd::ostringstream << file.rdbuf(). register_commands: builds acli::Menu("marketdata")submenu with one command,import: usagemarketdata import [--file <path>] [--fixings <path>] [--source <tag>].process_import:parse_argswith flagsfile,fixings,source(allrequires_value=true, empty default).- Reject if both
fileandfixingsare empty (nothing to import) — usage message. - Reject if not
session.is_logged_in(), matching bundles_commands. - Read each non-empty path via
read_file; fail loudly (path in the message) if a read fails. - Build
import_market_data_requestwithmarket_data_content,fixings_content,source(from the flags), send viado_auth_request<import_market_data_response>usingreq.nats_subjectand a generous timeout (mirrorpublish_timeout's 5 minutes — imports can be large). - On failure:
fail(out) << result->message. - On success: print
series_count=/=observation_count=/=fixing_countandresult->message.
- Includes
- Register the command in
projects/ores.shell/src/app/repl.cpp: add#include "ores.shell/app/commands/marketdata_commands.hpp"(alphabetical, afterlei_commands.hpp) andmarketdata_commands::register_commands(*root, session_);next to the otherregister_commandscalls. - Link the new dependency: add
ores.marketdata.api.libto thePRIVATElist intarget_link_librariesinprojects/ores.shell/src/CMakeLists.txt(sources are globbed, so no other CMake change is needed for the new .cpp). - Verify by building
ores.shell, logging in, and runningmarketdata import --file external/ore/examples/Legacy/Example_56/Input/market.txt --source ore.referenceagainst a running environment; confirmseries_count=/=observation_countmatch expectations and thatmarket_observationrows land taggedsource = "ore.reference". - Document: add a recipe under
doc/recipes/shell/marketdata/via thedoc-add-recipeskill, following the existingores.shellrecipe convention (org metadata drawer,ores-shellbabel block with the literal REPL session,#+RESULTS:). - Once this task is DONE and the story closes, resume the blocked task Import the 2016-02-05 ORE reference vintage using this new command.
Notes
PRs
| PR | Title |
|---|---|
| #1436 | [shell,marketdata] Add ores.shell market data import command |
Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | Consider a service-level test for modified_by/performed_by/change_reason_code | import_service.cpp | Declined | No other *_commands.cpp/import_service.cpp has unit tests today; noted as low-priority follow-up. |
| 2 | Usage-check runs before is_logged_in() check, ordering inconsistency | marketdata_commands.cpp | Declined | Cosmetic; other commands vary on this ordering too. |
| 3 | 5-minute import timeout is more generous than ImportTradeDialog's actual (unspecified/30s) default | marketdata_commands.cpp | Declined | Longer server-side wait is strictly safer for large files; GUI's 30s default being under-provisioned is a separate, pre-existing concern. |
| 4 | –file= (empty value) indistinguishable from omitting –file | marketdata_commands.cpp | Declined | Cosmetic; same ambiguity exists in other commands' flag parsing. |
Result
Shipped marketdata import [--file <path>] [--fixings <path>]
[--source <tag>] (marketdata_commands.{hpp,cpp}), following the
plan exactly. Verified end to end against a freshly-provisioned
Barclays tenant:
Academy/TA002_IR_Swap:✓ Imported 4 series, 52 observation(s), 0 fixing(s)(this vintage'sfixings.txtis empty, so0is expected).MarketRisk/HistSimVar:✓ Imported 217 series, 7172 observation(s), 904 fixing(s)— real fixings import confirmed working.Legacy/Example_56(2016-02-05, the vintage the blocked sibling task actually needs): fails deterministically on a Postgres unique- constraint violation inmarket_observations. Confirmed this is a pre-existing =import_service=/series-key-decomposition bug, not a shell-command bug — captured separately at Full market.txt import hits duplicate observation key.
Along the way, found and fixed two real import_service bugs
surfaced by this command's first non-interactive, full-file
exercise (not caught in review because the mechanism was previously
only exercised for a handful of curated quotes):
change_reason_codewas hardcoded to the literal"IMPORT", which doesn't exist inores_dq_change_reasons_tbl— nowdq::domain::change_reason_constants::codes::external_data_import.modified_by=/=performed_bywere hardcoded to"import", which fails the account/service-role validation trigger — nowctx_.actor()=/=ctx_.service_account(), matching every other service.
Documented via three recipes under doc/recipes/shell/marketdata/
(one per tested vintage, tangled into
projects/ores.shell/scripts/library/marketdata/), including an
honest "currently broken" recipe for Example_56 linking the
tracking capture rather than hiding the failure.
Follow-up: resume Import the 2016-02-05 ORE reference vintage once the duplicate-key bug is fixed.