Task: Complete the ores.shell synthetic market data commands

Table of Contents

This page documents a task in the Implement synthetic data generation against the new market data model story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

Give the shell everything a person needs to run synthetic market data end to end, so none of it requires the Qt client or code. The operations are: set a feed up (author the collection, the feed config, and the rows that make it work, an FX feed's GMM components or an IR feed's template entries and process parameter values), list the feeds and their state, start and stop a feed or a whole folder, validate a feed's vintage data, preview what a feed would produce, and watch it live. Watching is marketdata stream, which takes an oresmd URI, subscribes to the republished tick stream the marketdata service already produces for the caller's tenant and party, and either prints each tick to the screen until Ctrl-C or writes them to a CSV file. That is the fastest way to prove a synthetic feed is ticking and its ticks reach a consumer as themselves, because a bound synthetic feed's observations are republished on marketdata.v1.tick.<tenant>.<party>.<ore_key>.

2. Coverage today, so the gaps are explicit

  • Exists. Generated per-entity menus for the synthetic entities (market_data_generation_config, fx_spot_generation_config, gmm_component, ir_curve_generation_config, ir_curve_template_entry, ir_curve_generation_config_process_parameter_value, folder, yield_curve_process_type, yield_curve_process_parameter_definition), each with the usual list, get, put and delete. A hand-written synthetic menu with list folders, list feeds, start, stop and validate-vintage.
  • Missing. A porcelain way to set a feed up in one pass rather than assembling several raw entity writes by hand. State in the listing, so a person can see which feeds are enabled, auto-start and actually running, and which are partial. The preview operations the service already offers but the shell does not expose, synthetic.v1.fx_spot.simulate, synthetic.v1.ir_curve.preview_shape and synthetic.v1.ir_curve.simulate_paths. And marketdata stream.

3. Status

Field Value
State DONE
Parent story Implement synthetic data generation against the new market data model
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-07

4. Acceptance

Every operation below is reachable from the shell alone, with no Qt client and no code, and each states its result rather than failing silently.

  • Set a feed up. One porcelain flow authors a working feed: for FX, a collection and folder, the feed config and its GMM components; for IR, the feed config, its Curve Template entries and its process parameter values. The flow names the reference data it needs and refuses with the missing code rather than writing a half-configured feed.
  • List the feeds and their state. The listing shows, per feed, its asset class, source name, whether it is enabled and set to auto-start, and whether it is running now, so a partial feed is visible rather than absent.
  • Start and stop. A feed starts and stops by id, ORE key or source name, and a folder or collection cascades to every feed under it. Starting an already-running feed is idempotent and says so.
  • Validate and preview. validate-vintage reports each contributor's result as one summary, and the preview operations the service offers are reachable: synthetic.v1.fx_spot.simulate, synthetic.v1.ir_curve.preview_shape and synthetic.v1.ir_curve.simulate_paths.
  • Stream to the screen. marketdata stream <oresmd-uri> prints each tick as it arrives (observation time, the point oresmd URI, the value, the source) until Ctrl-C, then prints a summary count.
  • Stream to a file. marketdata stream <oresmd-uri> --csv <path> writes one row per tick with a header, still stopping on Ctrl-C, and the file is valid CSV for an empty run too.
  • Resolve the URI properly. The command resolves the oresmd URI to the datum's canonical ORE key and subscribes to the republished subject for the caller's tenant and party; an unparseable URI is refused with the codec's own message, not a raw subject error.
  • Stream a series. A series URI streams every point of that series, so a curve or surface can be watched without naming each point.
  • Stop cleanly. Ctrl-C exits without a stack trace, exit code 0 or the conventional 130, and a stream that receives nothing says so rather than hanging silently.

5. Plan

5.1. Where it goes

The operation surface lives in the hand-written synthetic menu in projects/ores.shell/application/src/app/commands/synthetic_commands.cpp, which already carries list folders, list feeds, start, stop and validate-vintage. Setup, state and preview extend that menu. The per-entity menus are generated and stay as they are.

marketdata stream is a hand-written subcommand of the marketdata menu in projects/ores.shell/application/src/app/commands/marketdata_commands.cpp, beside the hand-written import and export. The file already carries the parse_args-plus-process_* shape to follow.

5.2. How setup authors a feed

The generated per-entity commands already write each row; the gap is a flow that writes the rows a feed needs in one pass and refuses cleanly when a reference it needs is missing. For FX that is the collection, the folder, the market_data_generation_config, the fx_spot_generation_config and its gmm_component rows; for IR it is the config, its ir_curve_template_entry rows and its ir_curve_generation_config_process_parameter_value rows, whose definitions come from the process-type catalogue. The asset class supplies which rows a feed needs; the flow, its validation and its reporting are one.

5.3. How state and preview are surfaced

list feeds already asks synthetic.v1.feed_configs.list; the reply carries the running state, so the listing renders enabled, auto-start and running rather than only the names. The preview operations are already on the wire and need only a shell command each: synthetic.v1.fx_spot.simulate, synthetic.v1.ir_curve.preview_shape and synthetic.v1.ir_curve.simulate_paths.

5.4. How it streams

  • Resolve the argument with oresmd_uri_codec (the same codec the editor and the ingest use), then write the datum's canonical ORE key with ore_key_codec::write.
  • Subscribe on the shell's existing NATS session to market_tick_subject(tenant_id, party_id, ore_key) from ores.marketdata.api/domain/tick_subjects.hpp: the republished consumer subject marketdata.v1.tick.<tenant>.<party>.<key>. The tenant and party come from the session, so the command needs no extra arguments for them. The subscribe-print-unsubscribe shape already exists in ores.shell/application/src/app/commands/subscription_commands.cpp.
  • A series URI subscribes to the wildcard for that series' key prefix, so every point of the series streams.

5.5. Screen and CSV

Default writes one line per tick to the stream the REPL prints to; --csv <path> writes a header and one row per tick (observation_time, oresmd_uri, value, source). Ctrl-C is a SIGINT handler that unsubscribes, flushes and closes the file, prints the count and exits; the CSV is written flush-per-row or on exit so a killed stream still leaves usable data.

5.6. Testing

A command unit test with a stub session: a URI resolves to the expected subject, the CSV has the header and the rows, and an unparseable URI is refused with the codec's message. The end-to-end check is the manual scenario in this task: start a synthetic feed, run the command and watch it tick.

6. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
       

7. Notes

8. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

9. PRs

PR Title
   

10. Review

Comment summary File Decision Notes
       

11. Result

Merged in PR 2723 as commit cdb80e5645. Two commits: the shell surface, and the login fix it depended on.

Everything the acceptance asks for is reachable from the shell. marketdata stream resolves an oresmd URI, subscribes to the consumer subject the ingest loop republishes on, and writes ticks to the screen and to a CSV whose header precedes the first tick. synthetic list feeds, synthetic preview and synthetic setup complete the surface, with setup reusing the same put requests the generated add verbs send.

An independent review of the first revision returned FAIL with four blockers, and all four were real.

  • The series half of stream was built on a false premise. A series has no ORE key of its own, so the producer only ever names points, and a type's coordinates interleave with its identity fields, which means no NATS wildcard covers exactly one series' points. The command now watches the party's tick subtree and keeps the ticks whose series matches, testing membership with series_of. The earlier live proof had passed only because fx_spot has no coordinates at all, so series and point forms were identical.
  • plan_fx and plan_ir named the sub-config's own id as config_id instead of the owning container, orphaning every feed.
  • plan_ir left price_source and fixed_leg_payment_frequency_code unset, against a check constraint and a NOT NULL column, so every setup --kind ir failed on its first row.
  • Both folder rows used kind values outside the allowed set, and the failure landed after two writes, contradicting the claim that the plan is validated before the first write.

The tests that missed these called the planners directly. They now drive process_setup and use a rating series, which has two coordinates, so the filter is exercised rather than passing on a type whose series and point forms are identical.

Measured: ctest -R ores.shell passes 17/17 suites, ores.shell.application.tests holds 174 assertions across 40 cases. compass lint, the structure-note link check and the codegen drift check all pass.

Two follow-ups were found while proving this, and neither is this task.

  • A permission denial is reported as a decoding failure. A guarded read that a caller may not make is answered with an empty body and an X-Error: forbidden header, but the client only special-cases token_expired, so it tries to decode the empty body and reports "Failed to unpack msgpack data". This cost real time by sending the work after a wire-format defect that did not exist. The client should surface forbidden and unauthorized as such.
  • The environment has no account that holds the wildcard. No account holds *. demo_agent holds Member and nothing else: its thirteen SuperAdmin assignments were all created at 23:47:44 and soft-deleted at 23:48:02, so only the Member row is live, and every guarded read is refused. An earlier note here called the database corrupted by repeated seeding. That was wrong. The permission catalogue is correct, one row per code per tenant across thirty tenants; the rows that looked like duplicates were per-tenant rows and bitemporal versions of the same row. What is real is that the account lost its grants.

Emacs 29.3 (Org mode 9.6.15)