Task: Implement synthetic data generation against the new market data model
Table of Contents
- 1. Goal
- 2. Status
- 3. Acceptance
- 4. Plan
- 5. Measurements
- 6. Notes
- 6.1. Slice 1 done, 2026-10-06 (
f09d43f5bf). One =producer_subject(source, - 6.2. Slice 2 done, 2026-10-06 (
2e3ca06d17). Auto-creating afeed_bindingwas - 6.3. Slice 3 done, 2026-10-06 (
08b308b968). Aproducer_kindreference-data - 6.4. Slice 4 done, 2026-10-06 (
627f2386e1). The persist path is now proved - 6.5. Slice 5, fork disposition 2026-10-06. The inventory found 24 forks, 13 of
- 6.6. Slice 6, the bootstrap consumer check 2026-10-06. No code change is needed,
- 6.1. Slice 1 done, 2026-10-06 (
- 7. Test Scenarios
- 8. PRs
- 9. Review
- 10. Result
This page documents a task in the Implement synthetic data generation against the new market data model story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Prove and complete the path from a synthetic feed's tick to a row the new
market data model owns, and fix what does not hold. The feed already speaks
oresmd and can auto-create a feed_binding; the work is to show, end to end,
that its series is registered under the oresmd series identity, that each
observation lands under the right tenant, party, source and point identity,
that a reader can tell the data was generated rather than observed, and that a
consumer such as the curve bootstrap reads it and says where it came from.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Implement synthetic data generation against the new market data model |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-06 |
3. Acceptance
- A synthetic feed ticks and a market_series row exists whose oresmd_uri is the series identity the feed publishes, registered before any observation lands.
- Every observation the feed publishes reaches market_observation with its point identity in the oresmd URI, its source, and provenance that says generated rather than observed.
- The path is exercised end to end against a running service: start a feed, see the series and observations appear, and read them back by oresmd URI.
- The curve bootstrap consumes a synthetic feed's observations the same way it consumes market observations, and the resulting points record the synthetic provenance.
- The gaps found are fixed or captured as their own tasks, and the story states what a workspace-scoped or scenario feed would still need.
- IR and FX run one pipeline: the same feed abstraction, tick, binding, producer kind, ingest and persistence. Any remaining difference is the datum, the config fields or the process, and the fork inventory justifies every one of them.
4. Plan
4.1. What exists today, traced
- A feed publishes a
market_tick(oresmd_uriquote URI,value,observation_time,source) onsynthetic.v1.tick.<source>(or the sandbox prefix). ores::marketdata::service::app::feed_ingest_loopsubscribes to every producer's ticks.refresh()caches the enabledfeed_binding=s by source; each binding is one consumer. =persist()runs per binding:- derives the series URI with
oresmd_uri_codec::write(series_of(datum)); - creates the
market_seriesrow if the tenant+party+series URI has none, withseries_subclassfromseries_classifierandmarket_series_asset_classrows; - writes one
market_observationwithseries_id,observation_datetime,value,sourceand the pointoresmd_uri; - drops a tick from a source with no enabled binding, one warning per source.
- derives the series URI with
- The synthetic
feed_controllerauto-creates thefeed_bindingfor a bound feed, and a sandboxed feed never gets one.
So acceptance 1 and most of 2 already hold on paper. What is missing is proof and the provenance axis.
4.2. Gaps
- Nothing says generated.
market_series.derivation_kinddefaults toOBSERVED, and theOBSERVEDseed's own description says it is the default "for every existing producer (synthetic, vendor feeds)". The model therefore treats a synthetic series as observed, because observed currently means "directly published, not derived", not "real". There is no axis that says real versus generated.market_observationcarries onlysource. - No test proves the persist path.
app_tick_plan_testscovers routing, andfeed_controller_testscovers start/stop, but nothing drives a tick throughpersist()and reads back the rows.
4.3. Decisions
D1. Provenance is a second axis, not derivation_kind. Decided. derivation_kind
answers "was this published or computed?" (OBSERVED vs IR_CURVE_BOOTSTRAP,
CRM_DERIVATION), and a synthetic feed is genuinely published, so calling it a
derivation would break that meaning. Real-versus-generated is a separate axis
about the producer, carried by the feed_binding and readable from the
series an observation belongs to. Decided: a producer_kind reference-data
code table (VENDOR, SYNTHETIC) alongside derivation_kind, referenced by
feed_binding and market_series, listable in the UI and validated by the
seed rather than free text.
D2. One pipeline, asset class only where it is real. Decided. The generator pipeline is common to every asset class, and IR and FX must not have separate paths with spurious differences. An asset class may vary only where the difference is genuine: the shape of its datum, its config fields, and the process that produces its numbers. Everything around that — the feed abstraction, the tick, the oresmd identity, the subject, start and stop, the producer kind, the ingest, the provenance, the series and observation write — is one path parameterised by the config, not a branch per asset class. Any existing FX/IR fork in the pipeline is either justified against this rule or removed.
D3. Work in slices. Slice 0 is done: the fork inventory is at
evidence/asset-class-fork-inventory.org — 24 forks, 13 spurious, 11 genuine.
It changed the order, because it found that binding auto-creation is
unreachable from the live control plane (feed_controller::start(ore_key, GMM)
is FX-only with no production caller, and it is the only caller of
ensure_feed_binding), and that binding_mode is not expressible for IR, so a
sandboxed IR curve publishes on the ingested subject.
- 1. One producer subject, mode-aware for every kind: collapse the FX
synthetic_producer_subject(source, mode)and the IR hardcodedsynthetic_tick_subject(source), threadbinding_modeinto the IR build input and through the oneadd()/start()path. Fixes the IR sandbox bug; test both subjects. - 2. Binding creation on the one live start path, gated by
binding_mode, deleting the FX-only ad-hoc start. - 3. The
producer_kindfrom D1, on the shared binding and the shared persist path. - 4. The
persist()integration test: series and observation written, an unbound source dropped. - 5. Collapse the remaining spurious forks the inventory lists: the
feed_config_handlerkind resolution and duplicated start/stop, the duplicated vintage scan, the duplicated tick clock, the duplicated simulate/preview envelope, and the common config plumbing fields. - 6. The curve bootstrap consumer check.
5. Measurements
Every number this task claims about a tree, with the command that produced it
and the commit it was measured at. A number without both is not a measurement,
because a clean-up record describes a tree and trunk moves under a branch: a
consumer count, a gate's component count and a suite's size all change with no
code changing, and the reader of a bare number cannot tell which tree it came
from. Where the raw output matters, put it in an evidence/ file beside the
task and name that file in the Command column.
| What | Value | Command | Commit | ||||
|---|---|---|---|---|---|---|---|
| Forks inventoried | 24 (13 spurious, 11 genuine) | read evidence/asset-class-fork-inventory.org |
9c1c4af10e |
||||
| Slice 1 subject tests | 22 assertions in 7 cases pass | ores.synthetic.api.tests "[feed_factory]" |
f09d43f5bf |
||||
| marketdata tick-subject tests | 4 assertions in 3 cases pass | ores.marketdata.api.tests "[tick_subjects]" |
f09d43f5bf |
||||
| synthetic service tests, after DB recreate | suite passes (subject cases included) | ctest -R ores.synthetic.service.tests |
c8421b2d45 |
||||
| synthetic core tests, after DB recreate | 40/41, one fixture failure (missing ISO coding scheme) | ctest -R ores.synthetic.core.tests |
c8421b2d45 |
||||
| Slice 2 binding tests | service and api suites 2/2 pass | =ctest -R 'ores.synthetic.service.tests\ | ores.synthetic.api.tests'= | 2e3ca06d17 |
|||
| Slice 3 affected suites | 9/9 pass (refdata, marketdata, synthetic, shell) | =ctest -R 'ores.refdata\ | ores.marketdata\ | ores.synthetic\ | ores.shell.refdata\ | ores.shell.marketdata'= | 08b308b968 |
| Slice 3 codegen sweep | no file this branch touched would change | check_component_drift.py --sweep |
08b308b968 |
||||
| Slice 4 persist test | service suite 40 cases / 215 assertions pass, 18 in the 2 new cases | ctest -R ores.marketdata.service.tests |
627f2386e1 |
||||
| Slice 4 codegen sweep | no file this branch touched would change | check_component_drift.py --sweep |
627f2386e1 |
||||
| Fork disposition | 2 forks collapsed, 1 remaining group captured | evidence/asset-class-fork-inventory.org |
2887d018ef |
6. Notes
6.1. Slice 1 done, 2026-10-06 (f09d43f5bf). One =producer_subject(source,
mode)= in ores.synthetic/api/include/ores.synthetic.api/feeds/producer_subject.hpp
serves FX and IR; the IR build input and make_ir_curve_feed carry the mode,
and the auto-start path, feed_config_handler and folder_feed_control_handler
pass the configured mode instead of a default. IFeed gains nats_subject()
so the subject is assertable at the public seam. IR sandboxing is fixed.
The ores.synthetic.service.tests and ores.synthetic.core.tests binaries
refused to start while the local database schema fingerprint (0e5e94985734f233)
was behind the checkout (416d79146b6d63e3). The database was recreated on
2026-10-06 (fingerprint now 416d79146b6d63e3) and services restarted.
ores.synthetic.service.tests now passes, so the renamed feed_controller
subject cases run and pass. ores.synthetic.core.tests has one unrelated
failure from the fresh database, in feeds_fomc_curve_resolution_tests.cpp:92:
the country import inside its fixture cannot find the ISO_3166_1_ALPHA_2
coding scheme, so a populate step the fixture depends on is missing after
db recreate -k. That is an environment gap, not this change, and it needs
capturing as its own task before the service-level tests are dependable.
6.2. Slice 2 done, 2026-10-06 (2e3ca06d17). Auto-creating a feed_binding was
unreachable; the only caller of ensure_feed_binding was an FX-only
start(ore_key, GMM...) with no production caller. Both live paths, the
config-driven add() and the on-demand start(IFeed), now call one
bind_feed(source, mode, token) after releasing the mutex. It binds when the
mode is bound and a caller token exists, and does nothing for a sandboxed
feed or an internal start with no session. The marketdata round-trip sits
behind a one-method feed_binding_store seam, so the rule is tested without a
live service. The dead path and the should_ensure_feed_binding two-mode
contract it needed are deleted.
Open decision from slice 2: boot auto-start passes an empty token because no end-user session exists, so it still creates no binding. A binding for an auto-started feed must come from provisioning, or from a service-account identity decision about whose tenant and party the row lands in. Neither is settled and both are product calls, so they are recorded here rather than guessed at.
6.3. Slice 3 done, 2026-10-06 (08b308b968). A producer_kind reference-data
table holds VENDOR and SYNTHETIC. feed_binding carries it, defaulting to
VENDOR, and the synthetic binding store sets SYNTHETIC. The ingest stamps
the series it creates from the binding and leaves an existing series alone.
Everything is generated from the org models, including the SQL and the shell
surface for the new entity. The codegen sweep reports that no file this branch
touched would change. All nine affected suites pass, including the generated
producer-kind eventing test.
6.4. Slice 4 done, 2026-10-06 (627f2386e1). The persist path is now proved
against the database. feed_ingest_loop::handle_tick holds the whole of what
the loop does with an arrived tick and returns the drop reason the caller
observes, so a test reaches the real persist through the real path; on_tick
only decodes and delegates. A bound synthetic tick is driven through it and the
rows are read back by oresmd URI, with the series carrying producer_kind
SYNTHETIC and the observation carrying the point URI, source, series id and
value. A tick whose source has no enabled binding returns unbound_source and
leaves no series. The codegen sweep stays clean.
6.5. Slice 5, fork disposition 2026-10-06. The inventory found 24 forks, 13 of
them spurious. Each spurious one is now collapsed or captured.
- Collapsed. The producer subject and binding mode fork, one builder for
every kind (slice 1). The FX-only ad-hoc
start(ore_key, GMM...)and the vintage scan it alone duplicated, deleted in slice 2, which also removed the two-modeshould_ensure_feed_bindingcontract that existed only to protect its default argument. - Captured. The remaining spurious forks are the control-plane duplication
around the producer registry: the per-asset
feed_config_handlerstart and stop with kind resolved by probing the FX then the IR repository, the FX-shaped and triplicated vintage lookup, the duplicated tick clock, the duplicated simulate and preview envelope, the factory rebuilt at four call sites, and the common config plumbing fields repeated across the two generation-config entities. They are one coherent piece of work and are captured as Collapse the remaining synthetic control-plane forks, so this task can close without leaving them unrecorded. Every one is listed with its file and line inevidence/asset-class-fork-inventory.org.
6.6. Slice 6, the bootstrap consumer check 2026-10-06. No code change is needed,
and here is the evidence rather than an assertion.
- The ingest writes one
market_observationper bound consumer whatever the producer kind is, becausepersisttakes the binding and the datum only; nothing in it branches on the asset class or the kind. - The curve bootstrap reads observations through the same
market_observation_repositoryon the same table, so a synthetic series and a vendor series are the same shape to it. That is what makes the one-pipeline rule pay off here: the consumer needs no synthetic case. - The generated provenance is readable from the series: the slice 4 test reads
a series written by the ingest back from the database and asserts
producer_kind ="SYNTHETIC", and =market_seriescarries the field for every consumer to read.
7. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
8. PRs
| PR | Title |
|---|---|
| #2708 | [synthetic] Make the synthetic feeds first-class producers in the new market data model |
9. Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|
10. Result
The synthetic feeds now run the same pipeline as every other producer, and the path is proved end to end.
- Slices 1 to 4 are implemented, tested and committed, and the branch is in PR #2708.
- Slice 5's spurious forks are collapsed or captured; the one large remaining group is its own backlog task, Collapse the remaining synthetic control-plane forks.
- Slice 6 is evidenced rather than asserted: the ingest writes the one observation shape and the bootstrap reads that shape through the one repository, so no consumer needs a synthetic case, and the generated provenance is readable from the series a slice 4 test reads back.
- Two product calls are left open on purpose. Boot auto-start has no session,
so its binding must come from provisioning or a service-account identity.
And the shell
marketdata streamtask is backlog, which is the by-hand end-to-end check this work enables.