Story: Rework the market data feeds into a market data service with adapters

Table of Contents

This page documents a story in Sprint 28. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

1. Goal

ores.marketdata owns the catalogue, the mapping, the ingest path, routing and metering. Adapters, including the simulator, publish vendor-id ticks over NATS and answer a demand set. Every simulated feed belongs to the system tenant and the system party, a party takes a provider by subscription, and no part of the system behaves differently because a source is simulated.

2. Status

Field Value
State DONE
Parent sprint Sprint 28
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-24

3. Acceptance

The story closes on the tasks it finished:

  • Add the market data catalogue.
  • Key the series by item, provider and party.
  • Define the adapter contract.
  • Ingest by mapping and route.
  • Add subscriptions and routes.
  • Track adapter state and demand.
  • Meter usage events.
  • Add rate cards, rating and allocation.
  • Model composites and their rights.
  • Move the simulator to ores.marketdata.synthetic as an adapter.
  • Make generators system-owned and bundle-installed.
  • Add fault injection to the simulator.
  • Write the mapping inventory for each provider.
  • Describe the market data target architecture.
  • Fix the defects the first database run found.
  • Make the simulator native to each vendor.
  • The simulator reads its sources through marketdata.
  • The adapter fleet survives a failed start.
  • Plan the market data tenancy and the ticking guarantee.
  • Read routes through a role-scoped policy.

4. Tasks

Task State Start End Description
Add the market data catalogue DONE 2026-10-22 2026-10-22 Add providers, sources with a kind, vendor instruments with the minted flag, and mappings to ores.marketdata, with the source_kind vocabulary.
Key the series by item, provider and party DONE 2026-10-23 2026-10-23 Add the provider to the series key, and record the source id on each observation.
Define the adapter contract DONE 2026-10-23 2026-10-23 Add the announce, demand, tick and heartbeat messages and the ingress subjects.
Ingest by mapping and route DONE 2026-10-23 2026-10-23 Make the ingest loop map vendor ids, check quality, resolve the routed parties and store one observation for each.
Add subscriptions and routes DONE 2026-10-23 2026-10-23 Add the subscription with rights and the route to one source for each provider.
Track adapter state and demand DONE 2026-10-23 2026-10-23 Track each adapter's state, compute its demand set, and check its universe against the catalogue.
Meter usage events DONE 2026-10-23 2026-10-23 Record subscription intervals, demand intervals and delivery counts for each party.
Add rate cards, rating and allocation DONE 2026-10-23 2026-10-23 Add the rate card on a source, the rating step and the allocation of a source bill to parties.
Model composites and their rights DONE 2026-10-23 2026-10-23 Add the composite, its inputs and rule, the derived-right check, and the attribution of its use to its input providers.
Move the simulator to ores.marketdata.synthetic as an adapter DONE 2026-10-23 2026-10-23 Rename the synthetic component under market data and make the runner serve a demand set as an adapter.
Make generators system-owned and bundle-installed DONE 2026-10-23 2026-10-23 Move every collection, folder and generator to the system tenant and party and install them with the system bundle.
Add fault injection to the simulator DONE 2026-10-23 2026-10-23 Let the simulator drop, delay, repeat and corrupt ticks and reject ids on request.
Write the mapping inventory for each provider DONE 2026-10-23 2026-10-23 List the vendor instruments and mappings to install for Bloomberg, LSEG, ICE, Tradeweb and ORE.
Describe the market data target architecture DONE 2026-10-22 2026-10-22 Write the architecture pages, diagrams and journeys for the target state, and plan the implementation tasks.
Fix the defects the first database run found DONE 2026-10-23 2026-10-23 Fix the catalogue populate, the mapping date types, the as-of source id and a wrong test that a database run showed.
Make the simulator native to each vendor DONE 2026-10-23 2026-10-23 One simulated source for each vendor provider, publishing that vendor's own ids under a neutral source name, and no simulated source of the provider ore.
The simulator reads its sources through marketdata DONE 2026-10-23 2026-10-23 Restore ticking after the vendor-native move.
The adapter fleet survives a failed start DONE 2026-10-23 2026-10-23 Follow-ups of the review of PR 3687 and a restart defect.
Plan the market data tenancy and the ticking guarantee DONE 2026-10-24 2026-10-24 A plan that fixes the ownership of routes, the fragile control plane, the invisible failures and the missing end to end test.
Read routes through a role-scoped policy DONE 2026-10-24 2026-10-24 Replace the system-tenant widening of routes and subscriptions with a select-only policy for the marketdata service role.

5. Decisions

  • Closed at sprint 28 close with its 20 done tasks. The 16 open tasks and the original acceptance moved to Rework the market data feeds into a market data service with adapters — the remaining work in the product backlog inbox.
  • Market data has one owner, ores.marketdata. Adapters, including the simulator, are separate NATS services that meet it at four messages: announce, demand, tick and heartbeat. Marco decided this on 2026-10-22.
  • The mapping from a vendor instrument to an internal item is on the provider, not the source, so a live and a simulated source of one provider share it. Decided on 2026-10-22.
  • A series is keyed by item, provider and party. The source is a route and an observation records it for information. Decided on 2026-10-22.
  • A party has one active source for each provider. A source switch is audited and effective-dated. Decided on 2026-10-22.
  • Demand drives the adapters, ticks travel on core NATS, and a system bundle installs the catalogue and the simulator universe. Decided on 2026-10-22.
  • Vendor ids follow the vendor's notation. An id the system must invent carries the minted flag and a deviation note. Decided on 2026-10-22.
  • Composite licensing rights are modelled now: display, non_display and derived on the subscription. Decided on 2026-10-22.
  • The cost model records usage as events and prices it later with a rate card on the source. The platform pays the vendors and allocates the cost to parties, splitting a fixed fee equally among active subscribers. A simulated source has a simulated rate card and gives a shadow bill. Decided on 2026-10-22.
  • The simulator stops at the adapter seam. It imitates vendor data and failure behaviour, and it does not imitate vendor protocols. Decided on 2026-10-22.
  • ores.synthetic splits into ores.marketdata.synthetic and ores.trading.synthetic. Decided on 2026-10-22.
  • The simulator is native to each vendor and has nothing to do with ORE. A simulated source publishes the ids of its own provider, such as bbg_ticker for Bloomberg and ric for LSEG, and no simulated source of the provider ore exists. Decided on 2026-10-23.
  • A source name says which provider and which instance (bloomberg-a). It never says the kind, so no name, subject or id lets the system tell a simulated source from a live one. The kind shows in the kind column and as a badge. Decided on 2026-10-23.
  • The level of a synthetic price is calibrated from a free public source named in the generator config, as close to today as the source allows, and the screen shows when it was retrieved. The sources are the ECB euro reference rates and the Federal Reserve H.10 for FX, and the US Treasury par yield curve with the central banks' overnight rates for curves. The 2016 ORE vintage calibrates nothing. See Synthetic prices are calibrated from free public sources. Decided on 2026-10-23.
  • What public bodies published is loaded into one append-only table, published_quote, owned by the system tenant, with the chain from each row to its HTTP response and the terms it came under. The loading is independent of the calibration, and a generator config points at an item and a policy that a reader resolves. The item is in oresmd notation, the value is numeric as published, the raw bodies are kept, and a credential is resolved from a reference and never stored. Decided on 2026-10-23. Superseded on 2026-10-28, before the table was built: a public source is an adapter that publishes durable fixing ticks, and each publisher is a provider. See Public sources publish daily fixings through an adapter.
  • The story Subscribe tenants to provider feeds and prove the browser at scale describes feed bindings and overlaps this story. It needs a decision from Marco.

6. Out of scope

Emacs 29.3 (Org mode 9.6.15)