Task: Replace the asset-specific ticks with one market tick

Table of Contents

This page documents a task in the Rewrite the oresmd grammar as a hand-written codec checked against ORE story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

Every live market data feed, whatever its asset class, publishes and ingests one tick type keyed by the oresmd quote URI, and every feed takes its owner from one source.

2. Status

Field Value
State DONE
Parent story Rewrite the oresmd grammar as a hand-written codec checked against ORE
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-04

3. Acceptance

  • fx_spot_tick and ir_curve_tick are deleted; one modelled market_tick carries the oresmd quote URI, the value, the observation time and the source.
  • Every feed takes tenant, workspace and party from its feed binding; no tick carries an owner.
  • The ingest loop reads the URI with oresmd_uri_codec and classifies the datum with classification_key_of; it no longer rebuilds an ORE key from parts or reads a classification off the wire.
  • P02: marketdata.v1.tick is declared once, by the market_tick model, and every use of it, the republish subject builder included, references the generated constant.
  • The republished subject is marketdata.v1.tick.<tenant>.<workspace>.<party>.<key>, where <key> is the canonical ORE key of the tick's datum, lower-cased with its slashes turned to dots, written by one builder that both the ingest loop and the client use.
  • A tick finds its consumers by source through the enabled feed bindings of that source, for every feed; a tick from a source with no enabled binding is dropped with one warning per source.
  • market_tick.value is decimal text, as stored observations are.
  • Tenant provisioning creates bindings for the IR curve generation configs as it does for the FX spot ones.
  • The synthetic FX and IR curve feeds publish market_tick, and a live feed run stores observations for both.

4. Plan

The wire changes producers and consumers together, so this is one pull request of commits that each build:

  1. The tick. Model market_tick in ores.marketdata.operations, with :subject: marketdata.v1.tick, and write the one republished-subject builder in the api against its generated constant.
  2. The binding. A binding maps a source to the tenant, workspace and party that consume it. Its oresmd_uri and asset_class go: the tick names its own datum, and a curve source publishes many datums across several series. Its natural key becomes the party and the source.
  3. The ingest loop. One path for every tick: find the source's enabled bindings, read the URI with oresmd_uri_codec, classify with classification_key_of, store the observation, republish under the builder's subject.
  4. The producers. The synthetic FX and IR curve feeds publish market_tick. Tenant provisioning creates a binding for every feed, FX and IR alike.
  5. The deletions. fx_spot_tick, ir_curve_tick, their JSON I/O, the two copies of the subject function, and the FX-only client subscription, which becomes a subscription to market_tick.
  6. Proof. The fleet runs both synthetic feeds, and observations from both arrive under their datum URIs.

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
Change size 55 files, 549 insertions, 1,495 deletions git show --stat c87a9789b9
Test suites, recreated database 90 of 90 passed compass test run c87a9789b9
Bindings after ACME provisioning 34 sources (13 FX, 21 IR curve), 5 parties each compass sql, current feed bindings by source c87a9789b9
Sources ticking within five minutes of the feed start 13 FX spot, 20 IR curve compass sql, observations by instrument and source c87a9789b9
New series, classified from the datum 65 FX spot (fx, spot), 300 IR swap (interest_rates, yield) compass sql, series by instrument, subclass and asset class c87a9789b9

6. Notes

6.1. The live run

The fleet ran on a recreated database with the ACME demonstration profile, whose last step starts the market feeds. Every FX source and 20 of the 21 curve sources ticked, and both kinds were stored through the one path. The remaining curve, usdsofrfomc, published nothing: its first pillar ended on the spot date, and make_pillar_quote_key refused it, failing the whole batch. The cause is older than this change: tenant provisioning drops the FOMC schedule, so every meeting tenor resolves to spot. Provision the FOMC meeting schedule into every tenant fixes it.

6.2. Deleted rather than ported

fx_spot_subscription and the client's subject_helpers.hpp had no production consumer, so they went instead of being moved to market_tick.

The model and its reasons are in the market tick investigation. One point it leaves to this task:

  • The feed binding stores an asset_class that its oresmd_uri already implies. Derive it from the URI or drop it.

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

8. PRs

PR Title
#2607 [marketdata] Replace the asset-specific ticks with one market tick
#2602 [agile] Investigate the market tick data model

9. Review

# Comment summary File Decision Notes
1 PR 2602: the republish line references look stale. investigation Accepted They were correct at 2a000020ea (the two js_publish calls); the document now names the calls and functions instead of lines.
2 PR 2602: the tick design has no rule for the republished subject's suffix. investigation, task Accepted The suffix is the datum's canonical ORE key, lower-cased with dots, written by one builder; an acceptance line says so.
3 PR 2602: how a binding is found, and what happens with none. investigation, task Accepted By source, as FX does today, for every feed; no enabled binding drops the tick with one warning per source.
4 PR 2602: settle the value type before the model is written. investigation, task Accepted Decimal text, as stored observations are.
5 PR 2602: the task's Status still shows template text, and #+pr names the investigation PR. task Not changed The task is in BACKLOG until its work starts; compass pr create adds the implementation PR as another row.
6 PR 2602: no step creates bindings for the IR feeds; is source unique? investigation, task Accepted Provisioning creates them as it does for FX; a source names a producer uniquely within a tenant.
7 PR 2607: classifier_for returns a reference into a map that refresh() clears on another thread. feed_ingest_loop Accepted The map holds shared classifiers, and the caller keeps its own reference while it classifies.
8 PR 2607: the set of unnameable URIs is never cleared. feed_ingest_loop Accepted refresh() clears it, as it clears the unbound sources.
9 PR 2607: one series read per tick and binding. feed_ingest_loop Not changed The old paths read the series the same way, so it is not a regression. A series cache is a measured change for later.
10 PR 2607: the stats count a tick before it is stored. feed_ingest_loop Accepted A tick counts after its observation is stored.
11 PR 2607: feed_bindings changes with no migration. SQL Not changed No deployed database exists; the database is recreated, and the design decision allows no backwards compatibility.
12 PR 2607: the value is stored without a check. feed_ingest_loop Accepted A tick whose value is not a number is dropped, with one warning per source.
13 PR 2607: no tests for the new on_tick behaviour. tests Not changed on_tick needs NATS and a database; the test task, Close the marketdata test coverage, owns it and its notes say so.

10. Result

A live tick is one type for every feed and asset class: market_tick, the oresmd quote URI, the value as decimal text, the observation time and the source. Every feed takes its owners from its feed binding, which now maps a source to a party and nothing else, and the ingest loop has one path that classifies a new series from its datum, as the file import does. P02 holds for marketdata: marketdata.v1.tick is declared once, by the model, and the republished subject and both stream wildcards are built from it. On a recreated database, 90 of 90 suites passed, and a live ACME run stored FX and IR curve ticks from 33 of 34 sources; the one that published nothing fails on a pre-existing pillar defect, recorded as a capture.

Emacs 29.3 (Org mode 9.6.15)