Task: Move the stored oresmd identity and coordinate

Table of Contents

This page documents a task in the Make the oresmd URI grammar explicit and typed story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

An observation's coordinate is the canonical oresmd form of the datum, not the registry's ORE-form point. The row and the URI name the same coordinate in the same syntax, the registry's `point_id` is derived where a key must be projected, and a migration rebuilds both the series identity and the coordinate from the producer's own key.

2. Status

Field Value
State DONE
Parent story Make the oresmd URI grammar explicit and typed
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-01

3. Acceptance

  • One canonical coordinate form exists in the URI and in the database, and the row and the URI agree without translation.
  • The registry's point_id is derived for key projection and no longer written to the observation.
  • A migration re-derives the series identity and the coordinate from market_observations.key, and reports every row whose key is null.
  • The full corpus import stores every market payload, including both surfaces of a lognormal and normal vol pair.

4. Plan

4.1. The coordinate the row stores

The observation stores the datum's oresmd URI, and the series stores the same URI with its declared coordinates removed. Both are oresmd URIs, so the row and the URI agree by construction and a reader needs no translation table: the datum URI is the series URI plus the coordinate keys.

The alternative – storing only the coordinate keys, such as tenor=5y – was rejected because the consumers that must rebuild an ORE key need the whole identifier, and rebuilding it from a series id plus a key-value string is a translation of exactly the kind this column exists to remove.

The column is named oresmd_uri, the same name the series table gives its own, because it holds the same kind of value. The registry's ORE-form point_id stays in ores.ore, where the key grammar belongs, and is read from the decomposition where a key is projected.

4.2. The order of the work

  1. Rename the column in the two models (market_observation, observation_lineage) and regenerate the SQL and the repository entities. The compiler then names every C++ site.
  2. Teach the import to write the datum URI, which is to_uri() of the identifier it already builds, and to leave the series on to_series_uri().
  3. Move the consumers that want an ORE-form point to derive it: the curve pillar reader and the republish service both hold, or can hold, the datum URI. A rewrite of the key from the row is from_ore_key(key) for an imported row and with_point(series, ...) for a bootstrapped one.
  4. Write the migration. market_observations.key holds the producer's key, so the datum URI and the series URI are re-derived from it. A row whose key is null cannot be re-derived and is reported, not guessed.
  5. Rebuild the database, and make the corpus walk green – it is the unit's acceptance, and the two surfaces of a lognormal and normal vol pair are the case that used to collide.

4.3. What stays

series_key_registry::point_id, market_datum::point_id and the registry's shape table are untouched: they are ORE's grammar, not ours, and this unit removes only the copy of it that was being written into the observation.

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
Whole-corpus import walk Green: 482 assertions in 1 case ores.marketdata.core.tests "[corpus-full]" 61f544d091
Marketdata suites 4 of 4 green ctest -R 'ores.marketdata' 61f544d091
Whole tree 89 of 89 ctest suites green, 1,945s ctest -j 4 b0117e781c
Database rebuilt from this tree Schema 0.0.25, built at the branch head compass db recreate -y -k 61f544d091
Observations the walk stored 1,119,661 rows, 2,977 series select count(*) on the two tables 61f544d091

6. Notes

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

8. PRs

PR Title
#2497 [marketdata] Store the datum's oresmd URI as the observation's coordinate

9. Review

Comment summary File Decision Notes
       

10. Result

The observation's coordinate is the datum's canonical oresmd URI, and the whole-corpus import stores every market payload.

Delivered:

  • market_observations and observation_lineages carry oresmd_uri instead of point_id. The row and the URI name the coordinate in the same syntax: the datum URI is the series URI with the observation's coordinate keys left in, so a reader needs no translation between them.
  • The import writes it from the identifier it already builds. The feed ingress and the curve republish write the same projection, and the curve pillar reader finds a pillar by the datum URI rather than by decomposing the key.
  • The two synthetic feeds and the DQ publish bridge move with it: the DQ artefact carries the datum URI, and the bridge copies it into the observation.
  • The registry's ORE-form point_id is untouched and is now derived where a key is projected. The curve republish reads the point back off the datum URI.
  • The migration renames the column and reports the two populations an operator needs: rows still spelling an ORE point, which the next import rewrites, and rows with no producer key, which no import can re-derive. SQL cannot call the grammar, so the re-derivation itself is the import – which is what the corpus walk below measures.
  • The whole-corpus walk is promoted out of the hidden [.corpus-full] tag. It takes about twenty minutes, and it is the acceptance the story was written for, so it runs by default and the scheduled job carries it.

What is verified:

  • The whole-corpus import walk: 482 assertions in 1 case, green. It used to fail on its 32nd file with a duplicate observation key; it now imports every market payload and reads the values back.
  • The four marketdata suites, and all 89 ctest suites, green.
  • The database was rebuilt from this tree (compass db recreate -y -k) before the walk, so the schema the walk wrote into is the one this branch generates.

What is not done, recorded rather than glossed:

  • metric is still supplied at projection time for type=quote, as the previous unit recorded; credit's BASE_CORRELATION has no metric enum member.
  • The migration does not rewrite data. A database built before this change keeps the old ORE-form point in the renamed column until the corpus is re-imported, and the migration's report says how many rows that is. The house style for schema changes is recreation, and this follows it.

Emacs 29.3 (Org mode 9.6.15)