Task: Write the ORE key codec

Table of Contents

This page documents a task in the Rewrite the oresmd grammar as a hand-written codec checked against ORE story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

ore_key_codec reads ORE key text into market_datum and writes it back. It follows the structure of ORE's marketdatumparser.cpp and chooses each field by the token's value, so it assigns fields the way ORE does.

2. Status

Field Value
State DONE
Parent story Rewrite the oresmd grammar as a hand-written codec checked against ORE
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-02

3. Acceptance

  • Every key in the reference catalogue reads into the fields ORE assigns.
  • For every key ORE accepts, write(read(k)) = k=, or the canonical spelling where k is an alias.
  • Every key ORE rejects is rejected, with an error that names the form expected.
  • Dispatch is a switch on instrument_type, and the build fails on a missing case (-Werror=switch-enum on GCC and Clang, C4062 as an error on MSVC). No class hierarchy and no visitor.
  • The settlement segment, the case of free-text names and every optional token survive the round trip. No form uses a silent default.
  • The CDS running-spread form, the swaption payer/receiver flag, the commodity DEL and ATM/<type>/DEL forms, the equity MNY form and the 4-segment index CDS option form read into the right fields.

4. Plan

One reader and one writer per instrument type, each citing the ORE form it handles, split into one translation unit per asset-class family. Dispatch is a switch on instrument_type; typed access inside a reader uses a non-owning view over the datum. Test-first against the catalogue.

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
Documented forms checked 141: 121 read as ORE reads them, 19 refused as ORE refuses them, 1 refused on purpose ores.marketdata.core.tests "[ore_key_codec]" 864cef9c5a
Corpus keys checked 108,057, all read, written back and matched field by field ores.marketdata.core.tests "[ore_key_codec]" 864cef9c5a
Corpus check time 6.3 s (debug build; write reads each key back) ores.marketdata.core.tests "[ore_key_codec]" --durations yes ceaa39bb91
Codec test cases 10 cases, 45 assertions, all passed ores.marketdata.core.tests "[ore_key_codec]" ceaa39bb91
Missing dispatch case build fails: "enumeration value 'shape_profile' not handled in switch [-Werror,-Wswitch]" compass build ores.marketdata.core.lib with the case removed 864cef9c5a

6. Notes

The current readers choose fields by segment count. the 2026-10-02 review lists the mislabelled forms in section 1.

6.1. How the test compares fields

The catalogue prints ORE's datum through ORE's own inspectors. The test translates each datum into the same text: ORE's field names, ORE's defaults for a token the key leaves out, and ORE's printing of periods (12M prints as 1Y, 14D as 2W), reals (six significant digits) and derived dates (the as-of date plus the period, moved off a weekend). Each field ORE prints must have a translation. A field without one fails the test, so no field goes unchecked.

6.2. Keys ORE accepts that the codec refuses

ORE reads part of these keys and drops the rest. No datum can write such a key back, so the codec refuses it.

  • INDEX_CDS_OPTION with five tokens, where the last token starts with a number but is not a number, such as CDXIG/5Y/1Y. ORE's tryParseReal uses std::stod, which reads the 1 of 1Y, so ORE takes 1Y as a strike of 1. The documented "index term then expiry" form therefore works in ORE only for an expiry that does not start with a digit. The catalogue form is listed in the test's deliberately_refused table with this reason.

    This is a bug in ORE's OREData, not in QuantLib. ORE commit fb09012aa (2020-04-06) added the index term and states the rule in a comment: "If the last token is a number, we have 2) an expiry and a strike." Its tryParseReal had used std::stod since 3687c5743 (2018-02-26), and std::stod reads the leading number of 1Y and ignores the rest, so the rule never held for a period expiry. The consequence is wrong, not merely lenient: CDSVolCurve takes the quote as a 5Y expiry with the curve's default term, not a 1Y expiry on the 5Y index. ORE's own example data has no five-token key (44 have four tokens, 7,566 have six), which explains why the bug went unnoticed. This is the only place the parser chooses a form with tryParseReal. The codec keeps its refusal: reading the key as ORE intends would store a datum that ORE itself then reads differently.

  • The forms in the unit test the_codec_refuses_tokens_it_could_not_write_back: a cap/floor or swaption shift with a trailing flag, a base correlation tranche with an attachment point, and a shape profile with a sixth token other than DST.

6.3. Missing switch cases

The project builds with -Wall -Werror on GCC and Clang. -Wall turns on -Wswitch, which reports a switch on an enum that has no default and misses an enumerator. The dispatch switch has no default, so a new instrument_type without a case fails the build. The task acceptance named -Werror=switch-enum; that flag also reports switches that have a default, and the family readers keep a default that throws, so -Wswitch is the flag that fits. MSVC's equivalent, C4062, is off by default; ore_key_codec.cpp turns it into an error with a #pragma warning so it applies to that file only. No MSVC build ran on this branch, so the MSVC side is unverified.

6.4. Open point

QuantLib reads a signed period such as -1Y. The model's period does not. No catalogue key has a signed period, so the codec refuses one. Captured as The ORE key codec refuses a signed period.

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

8. PRs

PR Title
#2517 [marketdata] Write the ORE key codec against ORE's own parser

9. Review

# Comment summary File Decision Notes
1 Local review (minor): the readers throw refusal instead of returning std::expected. ore_key_codec.cpp Declined The exception never leaves the translation unit: read catches it and returns std::expected. Throwing keeps each reader a straight line of checks.
2 Local review (minor): the test cases do not log the system under test. datum_ore_key_codec_tests.cpp Declined The sibling catalogue tests do not log either, and a log line per corpus key adds 108,057 lines with no use.
3 Local review (minor): _MSC_VER appears outside ores.platform. ore_key_codec.cpp Declined It is a compiler check for one warning pragma, not a platform switch; ores.ore/core/src/domain/domain.cpp sets the same precedent.
4 Local review: a flag token outside its vocabulary had no test, because the catalogue has no such key. datum_ore_key_codec_tests.cpp Accepted Added a_flag_outside_its_vocabulary_is_refused.
5 Local review: the acceptance claimed MSVC catches a missing case, but nothing turned C4062 on. ore_key_codec.cpp Accepted A #pragma warning(error : 4062) under _MSC_VER. Not built on MSVC here.
6 PR 2517 round 1: type_tokens relies on its order matching the enum. ore_key_codec.cpp Accepted The table is gone. token_of is ORE's name, with a switch for the four types whose key spells it differently; type_from_token iterates the enum through token_of.
7 PR 2517 round 1: read catches only refusal. ore_key_codec.cpp Declined The parsers it calls return std::expected; anything else thrown is a programming error, and catching every exception would hide it.
8 PR 2517 round 1: exceptions as control flow cost a throw per refused key. ore_key_codec.cpp Declined Measured: the corpus runs in about 6 s in a debug build. Revisit if the codec sits on a hot ingest path.
9 PR 2517 round 1: write drops none fields, so a datum built directly can write a key that reads as another form. ore_key_codec.cpp Accepted write returns std::expected and refuses a series or a datum whose key does not read back equal. Test a_datum_whose_key_reads_as_another_form_has_no_key.
10 PR 2517 round 1: the MSVC pragma is untested. ore_key_codec.cpp Deferred Waiting for the next build with an MSVC leg.
11 PR 2517 round 1: the too-few-tokens message could say what is needed. ore_key_codec.cpp Accepted It now names the instrument type, the quote type and one field.

10. Result

ore_key_codec in ores.marketdata.core reads ORE key text into market_datum and writes it back. It reads every documented form and every corpus key as ORE's parser does, except one form where ORE drops part of a token (see the notes). It writes every key it reads back to the same text, with aliases in their canonical spelling. A missing dispatch case fails the build on GCC and Clang. The MSVC side is set up but not built on this branch.

Emacs 29.3 (Org mode 9.6.15)