Task: Cut the consumers over to the new codec

Table of Contents

This page documents a task in the Rewrite the oresmd grammar as a hand-written codec checked against ORE story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

Every reader and writer of market data identity uses the new codecs, and the tests prove values by key.

2. Status

Field Value
State DONE
Parent story Rewrite the oresmd grammar as a hand-written codec checked against ORE
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-02

3. Acceptance

  • import_service, ore_export_service, feed_ingest_loop, curve_republish_service, curve_pillar_reader and series_classifier use the new codecs.
  • The export writes each key from the stored datum URI. market_observations.key is gone.
  • A row cannot hold two identities. An FX pair the import reverses is reported, and the row holds the corrected pair only.
  • The corpus import test compares each value with the value under the same key.
  • The corpus selection includes the MD_* and FD_* files, and the tests share one tokeniser with production.
  • The database recreates, the whole corpus imports, and an export of the import equals the input after canonical spellings.
  • Every developer knows to recreate their database: the PR body names the step (compass db recreate -y -k, then re-import), and the sprint journal records the date the cut-over merged.

4. Plan

Every reader of a stored quote URI must move in the same change as every writer: a URI the new codec writes is one the old parser cannot read, and the seeds hold URIs too. So the cut-over is two PRs, each one green on its own.

  1. Readers and writers move together. The import, the export, the classifier's inputs, the feed ingest loop, the curve republish service, the curve pillar reader and the two synthetic feeds read and write quote URIs through ore_key_codec and oresmd_uri_codec. The SQL seeds that hold quote URIs (the FOMC series, the FX driver rates, the deposit rates and the synthetic curve configs' vintage series) are rewritten in the new form. The API gains one operation: a datum built from a series and its coordinate values, which the curve republish and the synthetic curve feed need where they used with_point. A reversed FX pair is reported and stored corrected. Fixings keep the old grammar, as the story's decision 9 allows: an index is not a quote.
  2. The raw key goes. market_observations.key is dropped through the model and regenerated; the corpus tests compare values by key, read the MD_* and FD_* files, and share the production tokeniser.

Recreate the database and re-import; write no migration.

The map of consumers, taken from grep of oresmd_parser, oresmd_projections, pillar_quote_key, market_data_identifier and series_key_registry outside the old grammar and its tests: import_service.cpp, ore_export_service.cpp, series_classifier.cpp, feed_ingest_loop.cpp/.hpp, curve_republish_service.cpp, curve_pillar_reader.cpp, fx_spot_feed.cpp, ir_curve_feed.cpp, feed_controller.hpp. The remaining hits are the old grammar's own API headers, a comment in ores.iam, and the test seed for the shape table.

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
Old-grammar calls left in production code the fixing paths of the import and the export only grep of oresmd_parser, oresmd_projections, with_point outside src/oresmd/ b1623fa8ab
Whole suite after recreating the database 89 of 89 passed (part 1) compass test run b1623fa8ab
Marketdata core suite 453 cases passed, 452 s, including the corpus import compared by canonical key and the export round trip compass test run -- -R marketdata b1623fa8ab
Spot pillar key ORE accepts IR_SWAP/RATE/USD/0D/1D/20260319 refused; .../0D/1D/7W and .../20260128/1D/20260319 accepted catalogue harness on the local ORE build b1623fa8ab
Whole suite, part 2, after recreating the database 89 of 89 passed before the split; core suite 592 s compass test run 4b9da9fbbf
Core suite after the split ores.marketdata.core.tests 167.87 s; ores.marketdata.core.corpus.tests 402.84 s; both passed compass test run -- -R "^ores.marketdata.core" 4b9da9fbbf
Slowest cases before the split whole-corpus walk 423 s; fixing walk 54 s; seed URI scan 18 s (std::regex) LastTest.log durations 4b9da9fbbf

6. Notes

See the 2026-10-02 review, section 2, "The ORE export echoes the input text" and "The corpus import compares values without their keys".

6.1. Part 1: the readers and writers

Done in the first PR. Each consumer now reads and writes quote identities through the two codecs, and the SQL seeds hold the new form.

  • A row holds one identity. The import stores the datum URI and the canonical key of that datum, never the file's own text. The export writes every key from the stored URI, so it no longer reads the key column at all.
  • A reversed FX pair. The FX convention checker is applied to the datum's two currencies, for an FX spot rate only, as before. The row holds the corrected pair, and the import reports it.
  • Classification. The rule table is keyed by the key's first token, its quote token and, for a correlation, its two indices. All three now come from the datum, so the shape table's decomposition is not read.
  • Fixings stay on the old grammar. An index is not a quote, and the story's decision 9 keeps fixings and curves on the richer contract. The export tells the two apart: a series URI the datum codec reads is a quote series.
  • Series plus point. The curve republish and the synthetic curve feed built a datum from a series and a pillar point through with_point. The API now has datum_at and datum_at_point for that, with no per-type code.

6.2. Two behaviour changes

  • The spot pillar key. make_pillar_quote_key wrote IR_SWAP/RATE/USD/0D/1D/20260319 for a pillar that starts at spot: a period start with a date end. ORE refuses it ("Expect swap quote with start/end as either periods or dates", checked with the catalogue harness), and the strict codec does too; the old grammar accepted a key ORE cannot read. A spot pillar is now 0D to the days its end lies after spot, such as 50D, which ORE reads as the same swap. This keeps one spot series across days. Writing the spot date instead would be ORE-valid too, but would make a new series every day.
  • Currency case. The old grammar upper-cased a currency, so FX/RATE/gbp/jpy became GBP/JPY. The datum model keeps every token as ORE wrote it, and ORE does not change it either, so the lower-case key is now its own series and no respelling is reported. An alias such as FX_SPOT is still respelled and reported.

6.3. Part 2: the raw key goes

  • market_observations.key is dropped through the model and regenerated: the domain type, the entity, the mapper, the protocol, the wire types and the table. The row's only identity is its datum URI. The import, the feed ingest loop and the curve republish stop writing the column, and the SQL publish from a DQ dataset stops copying it. The DQ staging table keeps its own key, which records the vendor's text for the dataset, not a stored observation.
  • The corpus tests compare values by key and read with the production tokeniser. Each file is read with parse_market_data or parse_fixings, the readers the import uses, and each stored row's key is written from its datum URI, so a value under another key or date fails.
  • The corpus selection takes the MD_* and FD_* files. The dated MD_* dumps are market payloads, which the catalogue's corpus extract already included, and the FD_* dumps are fixing payloads. A new case imports every fixing file and compares (date, index name, value).
  • The suite stays inside CI's limit. With the fixing walk and the MD_* files, ores.marketdata.core.tests ran 592 s, against the 600 s per-test limit CI applies on Linux. The whole-corpus walk is now its own ctest entry, ores.marketdata.core.corpus.tests, so each half has the full limit; both run by default. The seed URI scan no longer uses std::regex, which took 18 s in a debug build.

The cut-over's merge date is recorded in the sprint's Achievements.

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

8. PRs

PR Title
#2559 [marketdata] Drop the stored observation key and widen the corpus tests
#2545 [marketdata] Cut the quote readers and writers over to the new codecs

9. Review

# Comment summary File Decision Notes
1 PR 2545 round 1: the FX checker loads on a case-sensitive key prefix. import_service.cpp Accepted It now loads on the first datum that is an FX spot rate. A pair in a case refdata does not hold is left as written, not guessed at.
2 PR 2545 round 1: a quote URI the codec refuses is reported with the old parser's reason. ore_export_service.cpp Accepted The message names both reasons: the codec's and the fixing grammar's.
3 PR 2545 round 1: .value() after a successful read is unchecked. feed_ingest_loop.cpp, curve_republish_service.cpp Declined A datum the ORE key codec read always writes back, which the corpus test proves for every key; the tick path also catches and logs.
4 PR 2545 round 1: a spot pillar ending on or before spot makes a 0D or negative key. pillar_quote_key.cpp Accepted Refused with invalid_argument; test a_spot_pillar_that_does_not_end_after_spot_is_refused.
5 PR 2545 round 1: parse_value refuses empty text, which the URI codec now uses. market_datum.cpp Not changed The URI reader already refused an empty query value before, so nothing that read before stops reading.
6 PR 2545 round 1: parse_ore_key accepts an empty qualifier. feed_ingest_loop.cpp Accepted A comment says the codec, not the splitter, decides whether the parts make a key.
7 PR 2545 round 1: old-format URIs no longer read.   Noted The PR body names the recreate step; the journal records the merge date.
8 Local review, part 2: .journal.org is git-ignored, so it cannot carry the merge date to other developers. sprint.org Accepted Recorded in the sprint's Achievements instead.
9 Local review, part 2: the corpus test and the rest of the suite, run together, reached 592 s against CI's 600 s limit. tests/CMakeLists.txt Accepted The whole-corpus walk is its own ctest entry; the seed scan drops std::regex.
10 PR 2559 round 1: marketdata_market_observations_add_key_migration.sql still adds the dropped column. ores.sql/migration Accepted The migrations are a history, so a drop migration follows it, as for compute_results_drop_pgmq_msg_id; it runs and skips on a recreated database.
11 PR 2559 round 1: check CI time before merging.   Noted The build-and-test workflows run on a schedule, not on a PR, so the first CI timing comes after merge; it will be checked then.
12 PR 2559 round 1: the fixing walk runs in the main entry.   No change Measured: the main entry, including the 54 s fixing walk, ran 167.87 s.

10. Result

Every reader and writer of a stored quote identity uses ore_key_codec and oresmd_uri_codec, in two PRs. A stored observation holds one identity, its datum URI, and every ORE key is written from it; market_observations.key is gone. The SQL seeds hold the new URIs, and a test checks every URI they hold. The corpus tests read with the production tokeniser, compare values by key and date, cover the MD_* and FD_* dumps and every fixing file, and run in two ctest entries inside CI's per-test limit. Fixings keep the old grammar by design. Every developer recreates their database after each part merges; the sprint's Achievements record part 1's merge.

Emacs 29.3 (Org mode 9.6.15)