Task: Clean the marketdata code

Table of Contents

This page documents a task in the Rewrite the oresmd grammar as a hand-written codec checked against ORE story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

The hand-written marketdata code meets the structure and hygiene items of the standard.

2. Status

Field Value
State DONE
Parent story Rewrite the oresmd grammar as a hand-written codec checked against ORE
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-02

3. Acceptance

  • P02: moved to Replace the asset-specific ticks with one market tick, after the market tick investigation found that declaring the subject needs one tick type rather than two.
  • G08: ores.marketdata.api, ores.marketdata.client and ores.marketdata.core each have their namespace documentation header.
  • G02/H02: each of market_observations and market_fixings has one repository, and the survivor keeps the consumers both had.
  • The market_series table comment no longer describes the dropped columns.
  • Observation values convert through ores::platform::numeric::parse_double, not std::stod (curve_pillar_reader.cpp:59, curve_republish_service.cpp:128).
  • H03: no comment narrates edit history, no end-of-line comment remains, and comment volume follows the project rules, in marketdata's hand-written code and in the comments its models carry. The 14 shared templates that end every generated namespace with an end-of-line comment moved to Remove the namespace closer comments from the templates, because the change reaches every component.

4. Plan

Run the compass-code-review-comments pass last, over the whole component.

Three pull requests, one per verifiable unit:

  1. std::stod to parse_double, the market_series table comment, and G02: one repository each for observations and fixings.
  2. P02: the tick subjects declared in a model. The prefixes are written out in the ingest loop, both services' stream set-up and the synthetic feeds, and fx_spot_tick and ir_curve_tick are hand-written, so this is larger than one constant.
  3. H03: the comment-rules pass over the component and the templates that produce it.

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
C++ files in projects/ores.marketdata 285: 103 hand-written, 182 generated find for *.cpp and *.hpp, split on the AUTO-GENERATED FILE header 9fb7fc1eaf
End-of-line comments in hand-written files, before and after 33 before, 0 after, apart from 2 URIs in doc comments the scan misreads scratchpad/scan_comments.py 9fb7fc1eaf
Comments narrating history, story or task in hand-written files and models 10 rewritten the same scan, plus grep for story, task, now, no longer and deprecated 9fb7fc1eaf
Generated files with a namespace closer comment, across all components 2,873 from 14 templates git grep for the closer, filtered on the Template: header 9fb7fc1eaf

6. Notes

6.1. Part 3: the comment rules

The pass covered the 103 hand-written C++ files and the comments in the models.

  • Namespace closers lose their end-of-line comment: } // namespace becomes }, as the project's own files write it.
  • Ten other end-of-line comments moved to their own line above the code.
  • Comments that narrated history, stories or tasks now state the behaviour: "no longer splits keys", "now the series' oresmd identity", "the CRM story's architecture decision", "the config-creation task's own doc comment" and "today's synchronous publish" among them.
  • Comments that use "removed" or "added" to describe behaviour, such as "a field added to the enum and not here fails", stay.

The two review points PR 2597 deferred are in the same PR, as model edits regenerated into the repositories:

  • read_latest_for_series on observations and fixings reads the maximum timestamp once, as a static, as the other reads do.
  • read_as_of and read_as_of_buckets throw on a row with too few columns, a bucket with no ordinal, or an ordinal outside the requested buckets, rather than dropping the row. A dropped row would make a curve snapshot silently short a pillar.

6.2. Part 1: the two repositories

The generated market_observation_repository and market_fixing_repository survive. Their write makes a version claim per row, one query each, which an import of about 100,000 observations cannot afford, so the append the hand-written classes did is insert. It is a codegen feature, not pasted code: the :append_insert: flag in a model's SQL Flags drawer gives the repository insert(ctx, v) and insert(ctx, vector), so any append-only entity can use it. Codegen refuses the flag together with :current_state:, and test_append_insert.py covers it. The reads production used, read_latest_for_series, read_as_of and read_as_of_buckets, moved as custom methods too, declared in the models. The date-range read and remove-by-series had only test callers, so they went with the hand-written classes. The single-fixing test asserted only that nothing threw; it now reads the fixing back.

Correct the marketdata clean-standard record added the three namespace headers, so G08 already passes, and made subject_helpers.hpp use the api constant marketdata_tick_subject_prefix. What is left of P02 is declaring the subject in a model and generating that constant.

See the 2026-10-02 review, sections 3 and 4.

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

8. PRs

PR Title
#2630 [marketdata] Clean the marketdata code, part 3: comments and as-of reads
#2597 [marketdata] Clean the marketdata code, part 1: one repository each

9. Review

# Comment summary File Decision Notes
1 PR 2597: read_latest_for_series builds the maximum timestamp on every call. market_observation, market_fixing models Accepted Part 3: a static, as the other reads.
2 PR 2597: read_as_of and read_as_of_buckets skip a short row. market_observation model Accepted Part 3: they throw, and so does a missing or out-of-range bucket ordinal.
3 Part 3 local review: comment-only edits change no behaviour. hand-written marketdata Accepted Checked by the build, ctest ores.marketdata 5/5 and the marketdata drift check.
4 PR 2630: no test reaches the new throws. market_observation model Not changed A test needs the row mapping split from the query call; Close the marketdata test coverage takes it.
5 PR 2630: std::stoul on a malformed ordinal throws without context. market_observation model Accepted It throws the same runtime_error with the ordinal text.
6 PR 2630: the templates still write } // namespace. templates Not changed Hand-written files are not regenerated; Remove the namespace closer comments from the templates covers the generated ones.
7 PR 2630: one moved comment uses /// where its neighbour uses //. curve_republish_resolver.hpp Accepted It uses //. The comment in crm_ingest_bridge.hpp keeps ///, as its neighbours do.

10. Result

The marketdata code meets the items this task owns. Part 1 (PR 2597) left one repository each for observations and fixings, with a codegen append_insert flag for the bulk append, and moved observation values to parse_double. P02 was closed by the market tick task. Part 3 brought the hand-written code and the model comments to the comment rules, and made the as-of reads refuse a short row rather than return a curve short a pillar. The namespace closer comments that 14 shared templates write into every component are a task of their own.

Emacs 29.3 (Org mode 9.6.15)