Story: Implement synthetic data generation against the new market data model
Table of Contents
This page documents a story in Sprint 27. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
1. Goal
Make the synthetic feeds first-class producers in the new market data model. A feed already speaks oresmd and can auto-create a feed_binding, but nothing yet proves the whole path end to end: that its series is registered under the oresmd identity, that each observation it publishes carries per-point provenance and is marked generated rather than observed, that the ingest loop lands it in market_observation under the right tenant and source, and that a consumer such as the curve bootstrap reads it and says where it came from. This story proves that path, fixes the gaps it finds, and leaves the synthetic feeds able to drive the curve and surface work with data that can be plotted.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent sprint | Sprint 27 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-06 |
3. Acceptance
- A synthetic feed ticks and a market_series row exists whose oresmd_uri is the series identity the feed publishes, registered before any observation lands.
- Every observation the feed publishes reaches market_observation with its point identity in the oresmd URI, its source, and provenance that says generated rather than observed.
- The path is exercised end to end against a running service: start a feed, see the series and observations appear, and read them back by oresmd URI.
- The curve bootstrap consumes a synthetic feed's observations the same way it consumes market observations, and the resulting points record the synthetic provenance.
- The gaps found are fixed or captured as their own tasks, and the story states what a workspace-scoped or scenario feed would still need.
- IR and FX run one pipeline: the same feed abstraction, tick, binding, producer kind, ingest and persistence, with any difference justified as the datum, the config fields or the process.
4. Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Scaffold story: Implement synthetic data generation against the new market data model | DONE | 2026-10-06 | 2026-10-06 | Story scaffolding rides this task: documents, sprint wiring, and the scaffold PR. Close it before merging that PR. |
| Implement synthetic data generation against the new market data model | DONE | 2026-10-06 | 2026-10-06 | One pipeline for every asset class: prove the tick-to-row path, carry the producer kind, collapse the spurious FX/IR forks, and let the curve bootstrap read it. |
| Complete the ores.shell synthetic market data commands | BACKLOG | The full shell operation surface for synthetic data: set a feed up, list it and its state, start and stop it, validate and preview it, and stream an oresmd to the screen until Ctrl-C or to a CSV file. | ||
| Collapse the remaining synthetic control-plane forks | DONE | 2026-10-07 | The asset-class fork inventory lists the spurious forks left after the producer subject and binding gate were unified: the per-asset feed_config_handler, the triplicated vintage lookup, the duplicated tick clock, the duplicated simulate and preview envelope, the factory rebuilt at four call sites, and the duplicated config plumbing fields. Collapse them onto the one pipeline. | |
| Create the tick stream the service creates | DONE | 2026-10-07 | Task for: Implement synthetic data generation against the new market data model | |
| Give the synthetic control plane one per-kind registry | DONE | 2026-10-07 | 2026-10-08 | Seven spurious forks remain in feed_config_handler, folder_feed_control_handler, the vintage validity handler and the simulate and preview path. They all need the same missing seam: a per-kind registry that carries, beside the factory builder, the kind's config permission and a config_id to feed_build_input resolver. Collapsing them one at a time would grow a private registry per handler. |
5. Decisions
D1. One pipeline, asset class only where it is real. The generator pipeline is common to every asset class; IR and FX must not have separate paths with spurious differences. An asset class varies only in the shape of its datum, its config fields, and the process that produces its numbers. Everything around that is one path parameterised by the config.
D2. Provenance is a producer kind, not a derivation kind. derivation_kind
means published-versus-computed; real-versus-generated is a separate axis about
the producer, carried by the feed binding and readable from the series an
observation belongs to.
6. Out of scope
7. Result
Closed on the tasks' state, in the sprint's bookkeeping pass: every task is DONE with its work merged, and the tasks carry the outcomes. The owner had moved on before closing the story, so this close records the tasks' state rather than a fresh verification of the goal.