Task: Write the oresmd URI codec

Table of Contents

This page documents a task in the Rewrite the oresmd grammar as a hand-written codec checked against ORE story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

oresmd_uri_codec writes a market_datum as an oresmd URI and reads it back, with no code per family.

2. Status

Field Value
State DONE
Parent story Rewrite the oresmd grammar as a hand-written codec checked against ORE
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-02

3. Acceptance

  • The codec walks the datum's fields in schema order; it has no per-family branch.
  • The writer leaves out a field that holds none. The reader reads a missing key as none only where the schema row allows none, and refuses a missing required key, an unknown key or a repeated key.
  • The path is the row's subject field, percent-encoded with its case kept, and the query does not repeat it.
  • A type=quote URI follows the strict contract: every required field of its schema row present, an optional field present only when it holds a value, nothing else, no default, no partial form, one to one with the canonical ORE key. The reader refuses any type=quote URI that breaks it.
  • A type=series URI holds the identity fields only; the reader refuses a coordinate field in it.
  • For every datum in the reference catalogue: read(write(d)) = d=, write(read(u)) = u=, and the series URI parses.
  • Property tests generate datums from each schema row and each field's domain and round-trip them.

4. Plan

The value types' text forms come from the datum model task; this task writes the walk, the strict reader and the series URI.

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
Documented forms round-tripped 121 datums and their series; 119 distinct URIs for 119 distinct canonical keys (two forms are aliases) ores.marketdata.core.tests "[oresmd_uri_codec]" 21a0f0124d
Corpus keys round-tripped 108,057 datums and their series; 108,057 distinct URIs for 108,057 distinct keys ores.marketdata.core.tests "[oresmd_uri_codec]" 21a0f0124d
Free-text property test 4,540 of 4,560 generated variants exercised (20 have no ORE key), all round trip ores.marketdata.core.tests "[oresmd_uri_codec]" -s 21a0f0124d
Code vocabulary property test 259 variants, all round trip ores.marketdata.core.tests "[oresmd_uri_codec]" -s 21a0f0124d
Corpus check time 37.1 s (debug build; the reader also checks the spelling) ores.marketdata.core.tests "[oresmd_uri_codec]" --durations yes 21a0f0124d
URI codec test cases 10 cases, 64 assertions, all passed ores.marketdata.core.tests "[oresmd_uri_codec]" 21a0f0124d

6. Notes

Through today's URI, 7,254 corpus keys do not read back and 4,269 produce a URI the parser rejects. See the 2026-10-02 review, section 2.

6.1. The agreed shape

Agreed with the product owner on 2026-10-02: 

CDS/CREDIT_SPREAD/ACME/SNRFOR/USD/XR14/5Y
oresmd://credit/ACME?type=quote&instrument=cds&quote=credit_spread&seniority=SNRFOR&ccy=USD&doc_clause=XR14&term=5Y
oresmd://credit/ACME?type=series&instrument=cds&quote=credit_spread&seniority=SNRFOR&ccy=USD&doc_clause=XR14
  • The query keys are the schema's field names, so the codec needs no table from fields to URI names. The instrument and quote values are ORE's names in lower case (fx_spot, zc_inflationswap), which needs no table either.
  • A field that holds none is left out. The first proposal wrote every field, with an empty value for none. That rule came from the old grammar, where a missing key meant a default. In the typed model none has no default and the schema says which fields may be none, so the reader can tell a missing required key from none. Only one risk remains: a writer that forgets an optional field it meant to set. The codec is the only writer.
  • A series URI has type=series and holds the identity fields only.

6.2. What the schema gained

Each schema row now names its asset class and its subject field. Both are data that the URI needs and that classification will need when the consumers move over. A static_assert checks that every subject is an identity field of its row that is never none. The asset class names are the oresmd authorities ores.refdata's catalogue already maps: ir, fx, credit, equity, commodity, inflation, security, correlation, rating and shape_profile.

The subject is the currency for the rates rows, the reference entity or index for credit, the unit currency for FX, the index for inflation, the name for equity and commodity, and the security or contract for securities. Two rows needed a choice: a cross-currency basis swap uses ccy, the curve being built, not flat_ccy; a cross-currency fix-float swap uses fixed_ccy, the curve being built.

6.3. Strictness by the ORE key

A quote URI could name a datum that the schema admits but that no ORE key can write, such as a CDS with a doc clause and no seniority. The reader and the writer both call ore_key_codec::write, which reads its key back, so a quote URI is one to one with the canonical ORE key by construction. The cost is the corpus time above.

6.4. Encoding

Boost.URL writes the URI. The query uses space_as_plus = false, because a + is data in a strike or a free-text name, not an encoded space. Boost leaves = unencoded in a value; RFC 3986 allows it, and only the first = splits a key from its value. The test reserved_characters_are_percent_encoded_and_case_is_kept pins the exact text.

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

8. PRs

PR Title
#2522 [marketdata] Write the oresmd URI codec over the schema rows

9. Review

# Comment summary File Decision Notes
1 Local review (minor): the corpus round trip takes about 33 s, which adds to a suite that CI once timed out at 600 s. datum_oresmd_uri_codec_tests.cpp Declined The suite ran in 370 s before and 412 s after review round 1, inside the 600 s limit. Most of the time is the ORE key read-back that gives the one-to-one guarantee.
2 Local review (minor): market_data_urn.org still describes the old grammar. doc/knowledge/domain/market_data_urn.org Deferred The task "Rewrite the oresmd design record" owns it; the story's decision 6 states the new shape until then.
3 Local review: a value with reserved characters had no exact-text test, only round trips. datum_oresmd_uri_codec_tests.cpp Accepted Added reserved_characters_are_percent_encoded_and_case_is_kept.
4 Local review: a field could be named like a reserved query key and shadow it. datum_oresmd_uri_codec_tests.cpp Accepted Added no_field_is_named_like_a_reserved_query_key.
5 PR 2522 round 1: the reader accepts any query order, so several URIs name one datum. oresmd_uri_codec.cpp Accepted The reader now refuses any text but the writer's own spelling and names that spelling in the error, so stored URIs compare as text. This covers order and encoding variants such as %41. Story decision 6 says so.
6 PR 2522 round 1: a series with an empty subject writes a URI that does not read back. market_datum.cpp Accepted The datum model refuses empty free text in any field, so no datum can hold one. Tests free_text_cannot_be_empty and a_series_with_empty_text_has_no_uri.
7 PR 2522 round 1: lower and upper may duplicate a shared helper. oresmd_uri_codec.cpp Declined No shared helper exists. Eighteen files convert case on their own; captured as "ASCII case conversion is written out in eighteen files".
8 PR 2522 round 1: the return after the exhaustive switch in value_of may warn on some compilers. oresmd_uri_codec.cpp Declined Clang builds it clean under -Werror; the GCC and MSVC legs will show any warning.

10. Result

oresmd_uri_codec in ores.marketdata.core writes a datum or a series as an oresmd URI and reads it back by walking the schema row. Every documented form and every corpus key round trips as a datum and as a series, and gives back its canonical ORE key. The reader refuses each break of the contract that the refusal test lists. The old market_data_urn.org still describes the old grammar; the design record task rewrites it.

Emacs 29.3 (Org mode 9.6.15)