Market Data Architecture

One service, ores.marketdata, stands between the suppliers of market data and the parties that consume it. A supplier is who we contract with and connect to. A publisher is whose data it is. A session is one configured adapter at a supplier, and a feed binds a publisher to a session and a service. A live Bloomberg session, a replay session and the simulator all meet the service at the same four messages. The service maps supplier ids to internal items, checks them, stores each tick once for its feed, delivers the ticks to each subscribed party, and records what each party used so the platform can share the supplier cost. Nothing past the adapter seam knows whether a feed is live or simulated, and a person can always find out.

This page is the structure note for the whole subject. It gists and links and it carries no content its cluster already carries. It argues a reading order and says what the reader can do after each section.

The subject splits into six sections. The first says why the service exists and what rule keeps it honest. The second gives the model. The third follows a tick. The fourth compares real and simulated sources in detail. The fifth covers the governance of usage and cost. The sixth covers the surfaces a person uses and the stored data.

Two boundaries hold. This page does not carry the process mathematics, which is in Stochastic Processes. It does not carry the resolution path from a logical requirement to a concrete identifier, which is Market Data Requirements and Resolution.

These are the words of the model. The screens, the journeys and the names in the code use them. Each word has one meaning and one example.

The delivery layers nest: a supplier has sessions, a session offers services, a feed picks one service and one publisher on one session, and a party routes to a feed. The publisher is the other axis: a source of values, not a delivery path.

Read this section first. The remaining pages make sense only if the service owns the seam and the rule holds.

1. One service owns market data between the vendors and the parties. The service is the only place where vendor data becomes party data. This page shows the components.
2. No code path may behave differently because a source is simulated. The rule that decides every later design question: the source kind is data and never a branch.

After this section a reader is competent at saying what the service owns, and at spotting a change that would break the rule.

Read this section to know what the service stores and how the pieces join. The order goes from the provider to the party.

1. Every source belongs to a provider in the catalogue. The provider, the namespace of ids, and who installs the rows.
2. The source kind. A source is an adapter instance with a readable kind.
3. A vendor instrument is an id scheme, an id and a field. The scheme, the id and the field, in the vendor's own notation, and the minted flag.
4. The mapping belongs to the provider and not to the source. The link from a vendor instrument to an internal item, on the provider.
5. A quote series is keyed by item, provider and party. The key (item, provider, party), and the stream it points at. How the points are stored once for each source and read under the party's entitlement is in A source tick is stored once and entitled when it is read.
6. A party takes a provider's feeds through a subscription. The entitlement of a party to a provider.
7. Display, non-display and derived rights are modelled on the subscription. Display, non-display and derived.
8. A party routes to exactly one source for each provider. One active source for each (party, provider).
9. A composite attributes its use to the providers of its inputs. An item derived from several series, with its licence.

After this section a reader is competent at drawing the model on paper (see the diagram below) and at saying which table holds a given fact.

Read this section to follow one tick from the adapter to the party stream.

1. An adapter and the service meet at four messages. The four messages that make up the seam.
2. The adapter subjects. The subjects and the token rule for vendor ids.
3. The adapter lifecycle. The states of an adapter as the service sees them.
4. Demand drives what an adapter publishes. The set of ids each adapter serves, and why it is also a billing fact.
5. The identity an adapter runs as. Why an adapter knows no tenant and no party.
6. The ingest loop and the one pipeline. The one path from tick to store to party stream.
7. Quality checks run between mapping and storage and ignore the source kind. The checks between mapping and storage.
8. The market tick. The tick that the party receives.
9. The tick ingest pipeline: freshness first. The lanes, the ring and the batched writes that carry high tick rates.
10. A chart reads a series with one history call and one live subscription. Why the client makes two calls and merges them itself.
11. The history call returns every tick of a range in one reply, or refuses the range. What the one read of ticks accepts, returns and refuses.

After this section a reader is competent at tracing a tick from a vendor id to a stored tick and the party stream, at saying which stage drops a bad tick, and at saying which two calls a chart makes to read a series.

Read this section to know how the system handles real vendor data and synthetic data, and why the two differ only inside the adapter.

1. A live and a simulated source differ only inside the adapter. The side-by-side account.
2. Vendor adapters. What a live adapter adds behind the seam.
3. The simulator is one adapter plus the tools that author its generators. The simulator as an adapter plus its authoring tools, and its difference from ores.trading.synthetic.
4. The simulator stops at the adapter seam. How far the simulator goes: the data and the contract, and not the vendor protocol.
5. Generators belong to the system tenant and the system party. Why every generator is system-owned.
6. The synthetic runner. How demand starts and stops generators.
7. The generator factory and the kind seam, then Adding an asset class. How a new kind joins. A tick clock draws its gaps from an arrival process says when a generator's ticks arrive, as a random and bursty stream rather than a timer.
8. A system bundle installs the catalogue and the simulator installs its own universe. Where each definition lives and who installs it.
9. A source switch changes one route and nothing else. How a party moves between a simulated and a live source.
10. The data format of each provider: Bloomberg, LSEG, ICE, Tradeweb and ORE. The id scheme, the fields and the mapping of each, with the vendor's own documents.
11. The mapping inventory lists the vendor instruments each provider carries. The rows the bundle installs, which need a check, and what is not inventoried.
12. Synthetic prices are calibrated from free public sources. The free sources that set the level of the synthetic ticks, their terms, and what is still open.
13. Derivation of FX volatility quotes from historical fixings. Where no public source exists, an adapter of our own infers values from the stored fixings and publishes them under its own identity, so a vendor can replace it.
14. The notation of a provider is an independent class that roundtrips. The classes that read and write the ids of Bloomberg, LSEG and Demo, and the corpus that proves it.
15. Public sources publish daily fixings through an adapter. Each publisher is a provider, one adapter sends their fixings as durable ticks, and a feed starts from the latest one.
16. Market data ticks because ownership, demand and a smoke check agree. Why a provisioned tenant did not tick, the shortcomings it showed and the order of the work.

After this section a reader is competent at saying which parts differ between a live and a simulated source, and at planning a real adapter.

Read this section to know how the platform pays the vendors and shares the cost.

1. Usage is recorded as events and priced later. The events: subscription, demand and delivery.
2. A rate card on the source turns usage into a bill the platform shares. How vendors charge, the rate card on the source, rating and allocation, and the shadow bill of a simulated source.

After this section a reader is competent at saying what a party's bill is made of and where a rate is changed.

Read this section to drive the service and to read what it leaves behind.

1. Four areas of the Market data menu cover the service. The four areas of the Market data menu.
2. The shell operation surface. The shell commands.
3. The market simulator screens. The simulator screens.
4. The oresmd datum identity, then A quote series is keyed by item, provider and party, then Lineage and provenance. The identity of a datum and the stored result. A pillar names the market series it reads states the rule the identity implies: an oresmd URI is the runtime identity, and an ORE key lives only at the ORE XML boundary.
5. Snapshots and cuts. Snapshots and cuts.
6. ORE import and export. How an ORE market data vintage becomes observations.
7. Tracking market data identity from fixing to report. Why the chain is auditable through publication, and where it breaks.
8. Bootstrapping a curve: from quotes to a published set. The recipe, the quotes, the solve, the checks and the candidate.
9. Constructing a volatility surface: from quotes to a published set. The same construction for a surface, read after the curve because it differs only where a surface differs: a calibration in the place of a bootstrap, and a check of the result as well as of the recipe.

After this section a reader is competent at starting, inspecting and reading every part of the service from a screen or from the shell.

Read this section to author a generator. It belongs to the simulator and no other section depends on it.

1. The generation config collection and folders. The collection and the folder tree.
2. The FX spot generation config, then The GMM component, then The FX spot producer. One asset class end to end.
3. The IR curve generation config, then The IR curve template and its parameters, then The yield curve process type catalogue, then The IR curve producer. The second asset class.
4. A synthetic contributor publishes noisy observations of a latent surface. The third asset class, the FX volatility surface, read after the IR curve because its generator follows the curve's, with the one step a curve does not need: contributors between the state and the quotes.
5. The generator library. The process interface and the seeding rule.
6. Stochastic Processes. The mathematics.
7. Preview and simulate. The dry-run operations.

After this section a reader is competent at reading a generator config as a recipe and at saying which parts of a new asset class are config and which are code.

• The process mathematics, which is Stochastic Processes.
• The requirement to identifier resolution path, which is Market Data Requirements and Resolution. Its identifiers are requirement side and are not the stored datum. The requirement side identifier is Market Data Identifier, and the requirement side snapshot is Market Data Snapshot.

• Synthetic market data generation: approach. The reasoning behind the method per asset class and the cross asset constraints.
• Paper summary: Gaussian GenAI, synthetic market data generation. The source record that anchors the GMM choice.
• Scheduled Work Acting as a Tenant. Contradicts the adapter identity: scheduled work acts as a tenant, and an adapter acts as none.

• The data path — completes: how the ticks this cluster describes reach the store at one hundred thousand feeds.
• Curve Point Provenance — completes: the curve specific view of provenance.
• Cross-Rates Matrix — completes: the no-arbitrage spanning tree over driver rates.
• Spot rate blending — completes: the business case for the composite that blends spot rates.
• Anatomy of a Service — completes: how an adapter or an authority is built.
• Polymorphic types over NATS — completes: the typed per variant config pattern.
• Bind a report to a published curve — the reader's end of this cluster: the binding that makes a published set the market a report prices from, and the refusals that keep a candidate out of a run.
• Cuts — completes: the fixed world a report prices from once the published set is bound, and the deal population this binding still leaves unnamed.