Task: Model providers and feed scope

Table of Contents

This page documents a task in the Organise feeds by provider and scope story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

A provider is system reference data, seeded with the main vendors, exchanges, a broker and a generic provider, and published through DQ. A collection has a scope and may name its provider, every feed carries the synthetic mark, and the ACME reports' FX and IR currencies have feeds from one provider.

2. Status

Field Value
State DONE
Parent story Organise feeds by provider and scope
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-11

3. Acceptance

  • The provider catalogue is generated across every layer and published into the system tenant from the system_core bundle.
  • A collection's scope ties its tenant and party: a system collection is the system tenant's and its system party's; row-level security lets every tenant read it.
  • Every feed's source name starts synthetic., and the feed tables check it.
  • The ACME reports' FX spots and IR curves have feeds, set up by one synthetic feeds dataset that names its provider.

4. Plan

Model the provider catalogue in ores.synthetic and publish it through DQ; tie scope to tenant and party on the collection and widen row-level security for system collections; mark every feed synthetic; then replace the 2026 Realistic dataset with a Bloomberg feeds dataset that covers the ACME currencies, published the one way synthetic feeds are published.

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
       

6. Notes

  • The "synthetic theme" is renamed to the synthetic feeds dataset: artefact type synthetic_feeds, ores_synthetic_publish_feeds_from_dq_fn on synthetic.v1.ops.publish_feeds_from_dq, dataset codes synthetic.feeds.*, and the start_market_feeds step's dataset argument. It is the one way to set up synthetic feeds.
  • synthetic.feeds.bloomberg replaces the 2026 Realistic dataset, in the synthetic_bloomberg bundle: a USD spot for 26 currencies and an overnight-benchmark curve for 28, which covers the currency gaps of the ACME feeds capture. Each feed row names its provider (provider_code on the FX and IR artefacts); the collection names the provider when every row agrees. The 2016 ORE Samples dataset is unchanged beyond the rename.
  • The IR index families gained cdi, czeonia, buboron, telboron, tlref, camara, ibr and hibor, named as ORE's conventions name them, with matching overnight index conventions.
  • A first attempt published Bloomberg as a system collection through a second publish path. It was reverted: it duplicated the one path, and the generated repositories do not read the system tenant, so no tenant could bind to it.
  • The dead party provisioning plan (party_provisioning_plan.hpp, a hard-coded bundle list nothing called) is deleted.

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

8. PRs

PR Title
#3151 [synthetic] Model market data providers and feed scope, and set up the ACME feeds with Bloomberg

9. Review

# Comment summary File Decision Notes
1 Local review: misspell flags "existant" and "sucesso" i18n/locales/fr.ts, pt.ts Declined Correct French and Portuguese; neither line is in this change.
2 Local review: the provider DQ artefact uses a separate ores.dq model, a legacy pattern ores.dq.synthetic_market_data_provider.org Deferred Moved by Move the duplicated DQ models onto their owners.
3 Local review: system collections are readable in SQL only synthetic repositories Deferred Recorded in the Result and the provider catalogue page; the subscription task takes it up.
4 PR #3151: replace_all closes rows with current_timestamp, which falls before a same-transaction valid_from synthetic_publish_from_dq_create.sql Accepted Every close in the file uses clock_timestamp(); a pgTAP case fails on the old function and passes on the fix.
5 PR #3151: nothing in SQL ties a source name to the provider fx/ir feed tables Declined The feed tables have no provider column; the one publish path builds the name. The provider catalogue page says it holds by construction.
6 PR #3151: the provider validator passes everything during bootstrap ores_synthetic_validate_market_data_provider_fn Declined The providers ship in system_core, which system provisioning publishes before any tenant bundle; the feeds test publishes the catalogue first.
7 PR #3151: the system collection helper has no tenant filter synthetic_rls_policies_create.sql Accepted A comment states why: the scope check puts every system-scoped row in the system tenant.
8 PR #3151: the provider publish refusal uses raise exception, not ores_outcome_raise_fn synthetic_publish_from_dq_create.sql Declined No outcome code fits a publish aimed at the wrong tenant; it is an operator error, like the neighbouring dataset-not-found raises.

10. Result

Shipped:

  • market_data_provider in ores.synthetic, generated across every layer, with eight providers published from DQ into the system tenant.
  • Collection scope tied to tenant and party, the conflict guard kept, row-level security widened for system collections, and the synthetic mark checked on the FX and IR feed tables.
  • The "synthetic theme" renamed to the synthetic feeds dataset, the one way to set up feeds; a feed row may name its provider.
  • synthetic.feeds.bloomberg replaces the 2026 Realistic dataset: 26 USD spots and 28 overnight-benchmark curves plus the USD FOMC short end, with eight new IR index families and their conventions.

Not met: a system collection is readable in SQL but not through the services, because the generated repositories read the caller's own tenant alone, so no dataset publishes one yet. Tenants consume the Bloomberg feeds from their own party collection. Subscribe a tenant to provider feeds takes up system feeds a tenant subscribes to. The provider catalogue's DQ artefact follows a legacy pattern; Bring every DQ artefact table onto the canonical model moves it.

Emacs 29.3 (Org mode 9.6.15)