Story: Organise feeds by provider and scope

Table of Contents

This page documents a story in Sprint 27. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

1. Goal

Make a provider a system-level record, give every feed a system or test scope, seed the provider catalogue by provider and kind, and browse feeds through one tree by scope, provider and feed kind.

2. Status

Field Value
State DONE
Parent sprint Sprint 27
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-13

3. Acceptance

  • A provider is a system-level record with a code, a name, its kind, its logo and its asset classes, seeded with the main vendors and exchanges.
  • A feed has a scope: a system feed is owned by the installation under one provider, and a test feed is owned by a tenant or a party.
  • System feeds are organised provider, then feed kind, then the folders the provider's data needs.
  • Market data has one feed browser: a tree of System feeds by provider and Test feeds with run roll-ups, start and stop at any node, and the kind's list and feed page at its leaves.
  • Every path a feed is known by marks it as synthetic.

4. Tasks

Task State Start End Description
Model providers and feed scope DONE 2026-10-11 2026-10-11 A provider as system reference data, seeded with the main vendors and exchanges, and a scope on every feed: a system feed under a provider in the system tenant, or a tenant's or party's test feed.
Seed the provider feed catalogue DONE 2026-10-13 2026-10-13 System feeds per provider and kind, organised provider, kind, then the provider's folders, with provider-qualified source names, run once in the system tenant.
Build the feed browser DONE 2026-10-12 2026-10-13 One Market data feed browser: a lazily loaded tree of System feeds by provider and Test feeds with counts and run roll-ups, start and stop at any node, and the kind's list and feed page as its leaves.
Start the feeds after the parties exist DONE 2026-10-12 2026-10-12 The tenant run starts the market feeds as soon as the bundles are published, in parallel with the LEI import, because the step needs only publish_bundle. The step binds every party the tenant holds, so it binds the system party alone and the legal entities the import creates later get no bindings and store no prices. The step must also need provision_party.
Prototype the feed tree DONE 2026-10-12 2026-10-12 A throwaway web prototype of the feed tree screen before the feed browser is built: System feeds by provider and Test feeds, start and stop at any node with running roll-ups, search and scale to thousands of feeds, and leaves that open the existing FX and IR feed pages. It builds on the ores.qt Market Simulator and its desktop prototype.
Read system rows through the services DONE   2026-10-12 A tenant reads the installation's system rows through the synthetic service, which the feed tree and the feed overview need.
Build the Synthetic page and the feed overview DONE   2026-10-13 A Synthetic folder under Market data, and the feed overview: columns by scope, provider and feed kind with the counts and the running roll-up, and start and stop of a provider or a kind.
Build the feed tree DONE   2026-10-13 The tree of every feed by scope, provider and feed kind: opened a level at a time, counted by the service, searched by words, with start and stop at any folder, and a feed's page whose trail runs through it.
Give every feed a provider and an asset class, and start or stop a selection on the server DONE 2026-10-13 2026-10-13 The server side of the single Synthetic screen: providers, asset classes, filters and selection.
Make Synthetic one screen with a dashboard and a feed tree DONE 2026-10-13 2026-10-13 The web side: one screen with two tabs replaces the overview, the tree and the two lists.
Guard feed runs with a permission, tell the client only its own mistakes, and place a feed added by hand DONE 2026-10-13 2026-10-13 Closes the follow-ups of the Synthetic screen: a run permission, the raw error text, and the feed with no folder.

5. Decisions

The remainder is carried in Subscribe tenants to provider feeds and prove the browser at scale.

  • Two scopes, one feed shape. A system feed and a test feed are the same record with the same process and the same screens; the scope decides who owns it, who may write it, and who may read it. The FX price feed screen needs no second editor.
  • Providers are system data. A provider (Bloomberg, Refinitiv, ICE, CME, LSE, a broker) is reference data of the installation, in the system tenant, as the other shared catalogues are. Its feeds run in the system tenant, once, and publish to the bus once.
  • Consumption is a binding, not a copy. A tenant or party that wants a provider's FX spot does not get a copy of the feed; it gets a feed binding onto the provider's source, which the market data ingest already honours: the tick is stored once per enabled binding, under the binding's tenant and party. This is the consistent-world model the bindings were built for, and it is why A feed binding's existence claim is keyed on the ORE key alone and Name a feed by its oresmd URI come first: many parties will bind one source.
  • Organised by provider, then kind, then the provider's own folders. The folder hierarchy stays the mechanism, and the system tenant's root is System feeds with one folder per provider. Under a provider, a folder per feed kind (FX spot, IR curves, later surfaces and order books), then what the kind needs, such as a pair group or an exchange segment. A tenant's test feeds keep the party root and its collections.
  • One browser, kind screens as leaves. Market data gets a feed browser: a tree on the left (System feeds by provider, Test feeds), the chosen node's feeds on the right through the kind's list, and the kind's feed page when one is opened. The FX price feed list and page become that leaf, and the per-kind cards under Market data become entry points into the tree.
  • One way to set up feeds. Feeds come from a synthetic feeds dataset (DQ artefact type synthetic_feeds), published by one function into the party provisioning names; the "theme" was that dataset and is renamed to it. A feed row names its provider, and its source name follows the provider. Until the services read system collections, a provider's feeds are published as a party collection like any other.
  • Built for scale from the first screen. Order books and wide provider catalogues mean tens of thousands of feeds. The tree loads one level at a time and asks the server for each node's counts and run roll-up; every list is paged, searched and sorted on the server; start and stop act on a node through the folder cascade the service already has; the running read and the tick buffers are checked at that size. No screen reads all feeds.
  • Every path says the feed is synthetic. A simulated Bloomberg feed must never be read as Bloomberg. Every name a feed is known by starts with the synthetic mark: the source name (synthetic.bloomberg.fx.eurusd), the tick subject (synthetic.v1.ops.tick.synthetic.bloomberg.fx.eurusd), the folder path (Synthetic feeds › Bloomberg › FX spot), and the provider as a screen names it (Bloomberg (synthetic)). A real provider connection added later uses its own namespace and cannot collide.
  • Source names carry the provider. After the synthetic mark, a system feed's source is qualified by its provider and a test feed's by its party, so two scopes never claim one source.
  • A system collection is the system tenant's. It belongs to the system tenant and its system party, the party the super administrator works in, as every other shared catalogue does. The earlier null-tenant reading of system scope could hold no feeds, because a feed must share its collection's tenant.
  • One bound provider per pair, for now. The collection's conflict guard stays: one enabled, bound system collection at a time. So the first step creates the providers and binds one of them, chosen arbitrarily, as the authoritative source of a pair such as EUR/USD; the other providers' collections run sandboxed. The second step is a rate blending service that reads the providers' spots and owns the blended EUR/USD itself.

6. Open questions

  • Does a provider feed's process belong to the installation administrator only, or may a tenant tune a private copy of a provider feed as a test feed?
  • Is a tenant's subscription one binding per feed, or one per provider and kind with the feeds under it implied?
  • Do order books share the feed record and the process editor, or are they a kind with its own model (levels, depth, events)?

7. Out of scope

  • Real connections to Bloomberg, Refinitiv or an exchange. These are simulated providers.
  • Order book feeds themselves; this story makes the organisation and the browser ready for them.
  • Entitlements and billing of provider data per tenant.

Emacs 29.3 (Org mode 9.6.15)