Task: Complete the ores.shell synthetic market data commands
Table of Contents
This page documents a task in the Implement synthetic data generation against the new market data model story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Give the shell everything a person needs to run synthetic market data end to
end, so none of it requires the Qt client or code. The operations are: set a
feed up (author the collection, the feed config, and the rows that make it
work, an FX feed's GMM components or an IR feed's template entries and process
parameter values), list the feeds and their state, start and stop a feed or a
whole folder, validate a feed's vintage data, preview what a feed would
produce, and watch it live. Watching is marketdata stream, which takes an
oresmd URI, subscribes to the republished tick stream the marketdata service
already produces for the caller's tenant and party, and either prints each
tick to the screen until Ctrl-C or writes them to a CSV file. That is the
fastest way to prove a synthetic feed is ticking and its ticks reach a
consumer as themselves, because a bound synthetic feed's observations are
republished on marketdata.v1.tick.<tenant>.<party>.<ore_key>.
2. Coverage today, so the gaps are explicit
- Exists. Generated per-entity menus for the synthetic entities
(
market_data_generation_config,fx_spot_generation_config,gmm_component,ir_curve_generation_config,ir_curve_template_entry,ir_curve_generation_config_process_parameter_value,folder,yield_curve_process_type,yield_curve_process_parameter_definition), each with the usual list, get, put and delete. A hand-writtensyntheticmenu withlist folders,list feeds,start,stopandvalidate-vintage. - Missing. A porcelain way to set a feed up in one pass rather than
assembling several raw entity writes by hand. State in the listing, so a
person can see which feeds are enabled, auto-start and actually running,
and which are partial. The
previewoperations the service already offers but the shell does not expose,synthetic.v1.fx_spot.simulate,synthetic.v1.ir_curve.preview_shapeandsynthetic.v1.ir_curve.simulate_paths. Andmarketdata stream.
3. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Implement synthetic data generation against the new market data model |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-07 |
4. Acceptance
Every operation below is reachable from the shell alone, with no Qt client and no code, and each states its result rather than failing silently.
- Set a feed up. One porcelain flow authors a working feed: for FX, a collection and folder, the feed config and its GMM components; for IR, the feed config, its Curve Template entries and its process parameter values. The flow names the reference data it needs and refuses with the missing code rather than writing a half-configured feed.
- List the feeds and their state. The listing shows, per feed, its asset class, source name, whether it is enabled and set to auto-start, and whether it is running now, so a partial feed is visible rather than absent.
- Start and stop. A feed starts and stops by id, ORE key or source name, and a folder or collection cascades to every feed under it. Starting an already-running feed is idempotent and says so.
- Validate and preview.
validate-vintagereports each contributor's result as one summary, and the preview operations the service offers are reachable:synthetic.v1.fx_spot.simulate,synthetic.v1.ir_curve.preview_shapeandsynthetic.v1.ir_curve.simulate_paths. - Stream to the screen.
marketdata stream <oresmd-uri>prints each tick as it arrives (observation time, the point oresmd URI, the value, the source) until Ctrl-C, then prints a summary count. - Stream to a file.
marketdata stream <oresmd-uri> --csv <path>writes one row per tick with a header, still stopping on Ctrl-C, and the file is valid CSV for an empty run too. - Resolve the URI properly. The command resolves the oresmd URI to the datum's canonical ORE key and subscribes to the republished subject for the caller's tenant and party; an unparseable URI is refused with the codec's own message, not a raw subject error.
- Stream a series. A series URI streams every point of that series, so a curve or surface can be watched without naming each point.
- Stop cleanly. Ctrl-C exits without a stack trace, exit code 0 or the conventional 130, and a stream that receives nothing says so rather than hanging silently.
5. Plan
5.1. Where it goes
The operation surface lives in the hand-written synthetic menu in
projects/ores.shell/application/src/app/commands/synthetic_commands.cpp,
which already carries list folders, list feeds, start, stop and
validate-vintage. Setup, state and preview extend that menu. The per-entity
menus are generated and stay as they are.
marketdata stream is a hand-written subcommand of the marketdata menu in
projects/ores.shell/application/src/app/commands/marketdata_commands.cpp,
beside the hand-written import and export. The file already carries the
parse_args-plus-process_* shape to follow.
5.2. How setup authors a feed
The generated per-entity commands already write each row; the gap is a flow
that writes the rows a feed needs in one pass and refuses cleanly when a
reference it needs is missing. For FX that is the collection, the folder, the
market_data_generation_config, the fx_spot_generation_config and its
gmm_component rows; for IR it is the config, its ir_curve_template_entry
rows and its ir_curve_generation_config_process_parameter_value rows, whose
definitions come from the process-type catalogue. The asset class supplies
which rows a feed needs; the flow, its validation and its reporting are one.
5.3. How state and preview are surfaced
list feeds already asks synthetic.v1.feed_configs.list; the reply carries
the running state, so the listing renders enabled, auto-start and running
rather than only the names. The preview operations are already on the wire and
need only a shell command each:
synthetic.v1.fx_spot.simulate, synthetic.v1.ir_curve.preview_shape and
synthetic.v1.ir_curve.simulate_paths.
5.4. How it streams
- Resolve the argument with
oresmd_uri_codec(the same codec the editor and the ingest use), then write the datum's canonical ORE key withore_key_codec::write. - Subscribe on the shell's existing NATS session to
market_tick_subject(tenant_id, party_id, ore_key)fromores.marketdata.api/domain/tick_subjects.hpp: the republished consumer subjectmarketdata.v1.tick.<tenant>.<party>.<key>. The tenant and party come from the session, so the command needs no extra arguments for them. The subscribe-print-unsubscribe shape already exists inores.shell/application/src/app/commands/subscription_commands.cpp. - A series URI subscribes to the wildcard for that series' key prefix, so every point of the series streams.
5.5. Screen and CSV
Default writes one line per tick to the stream the REPL prints to; --csv
<path> writes a header and one row per tick (observation_time,
oresmd_uri, value, source). Ctrl-C is a SIGINT handler that unsubscribes,
flushes and closes the file, prints the count and exits; the CSV is written
flush-per-row or on exit so a killed stream still leaves usable data.
5.6. Testing
A command unit test with a stub session: a URI resolves to the expected subject, the CSV has the header and the rows, and an unparseable URI is refused with the codec's message. The end-to-end check is the manual scenario in this task: start a synthetic feed, run the command and watch it tick.
6. Measurements
Every number this task claims about a tree, with the command that produced it
and the commit it was measured at. A number without both is not a measurement,
because a clean-up record describes a tree and trunk moves under a branch: a
consumer count, a gate's component count and a suite's size all change with no
code changing, and the reader of a bare number cannot tell which tree it came
from. Where the raw output matters, put it in an evidence/ file beside the
task and name that file in the Command column.
| What | Value | Command | Commit |
|---|---|---|---|
7. Notes
8. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
9. PRs
| PR | Title |
|---|---|
10. Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|
11. Result
Merged in PR 2723 as commit cdb80e5645. Two commits: the shell surface,
and the login fix it depended on.
Everything the acceptance asks for is reachable from the shell. marketdata
stream resolves an oresmd URI, subscribes to the consumer subject the
ingest loop republishes on, and writes ticks to the screen and to a CSV
whose header precedes the first tick. synthetic list feeds, synthetic
preview and synthetic setup complete the surface, with setup reusing the
same put requests the generated add verbs send.
An independent review of the first revision returned FAIL with four blockers, and all four were real.
- The series half of
streamwas built on a false premise. A series has no ORE key of its own, so the producer only ever names points, and a type's coordinates interleave with its identity fields, which means no NATS wildcard covers exactly one series' points. The command now watches the party's tick subtree and keeps the ticks whose series matches, testing membership withseries_of. The earlier live proof had passed only becausefx_spothas no coordinates at all, so series and point forms were identical. plan_fxandplan_irnamed the sub-config's own id asconfig_idinstead of the owning container, orphaning every feed.plan_irleftprice_sourceandfixed_leg_payment_frequency_codeunset, against a check constraint and a NOT NULL column, so everysetup --kind irfailed on its first row.- Both folder rows used
kindvalues outside the allowed set, and the failure landed after two writes, contradicting the claim that the plan is validated before the first write.
The tests that missed these called the planners directly. They now drive
process_setup and use a rating series, which has two coordinates, so
the filter is exercised rather than passing on a type whose series and
point forms are identical.
Measured: ctest -R ores.shell passes 17/17 suites, ores.shell.application.tests
holds 174 assertions across 40 cases. compass lint, the structure-note
link check and the codegen drift check all pass.
Two follow-ups were found while proving this, and neither is this task.
- A permission denial is reported as a decoding failure. A guarded read
that a caller may not make is answered with an empty body and an
X-Error: forbiddenheader, but the client only special-casestoken_expired, so it tries to decode the empty body and reports "Failed to unpack msgpack data". This cost real time by sending the work after a wire-format defect that did not exist. The client should surfaceforbiddenandunauthorizedas such. - The environment has no account that holds the wildcard. No account
holds
*. demo_agent holdsMemberand nothing else: its thirteenSuperAdminassignments were all created at 23:47:44 and soft-deleted at 23:48:02, so only theMemberrow is live, and every guarded read is refused. An earlier note here called the database corrupted by repeated seeding. That was wrong. The permission catalogue is correct, one row per code per tenant across thirty tenants; the rows that looked like duplicates were per-tenant rows and bitemporal versions of the same row. What is real is that the account lost its grants.