Task: Prove the round trip preserves the data and validates

Table of Contents

This page documents a task in the Decide the XML strategy for ORE documents story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

Build the harness that makes every later migration verifiable. Round trip here means the values survive and the output validates; it does not mean the document is reproduced. Without this harness a family moves to the codec with no test that its data came with it.

2. Status

Field Value
State DONE
Parent story Decide the XML strategy for ORE documents
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-17

3. Acceptance

  • The harness reads the corpus, writes it, validates the output with xmllint against input.xsd, reads it back and compares.
  • The comparison is against the source values, not against the codec's own parse of the source, so a read-side drop cannot pass.
  • Any element the model did not claim fails the harness.
  • The same command reproduces the result on demand.

4. Plan

(Implementation strategy. Written when work starts; key decisions are distilled into the parent story's * Decisions at close, but the plan itself stays — it is the historical record of what we did.)

5. Plan

Two comparisons, because one is not enough.

  • The records. compare_records walks two records by the same table and names the first member they disagree about, as a path a reader can follow: calendaradjustment.Calendar[0].AdditionalHolidays.Date[0]. It compares presence bits, so a value that vanishes is "was lost" rather than an empty string that matches nothing, and it compares amounts as decimals and continuous quantities as doubles.
  • The documents. Two records agreeing only proves the writing keeps what the reading kept. A read-side drop would pass that. So the harness also reads the source and the writing with the document layer – no schema, no table, no record – and compares a value fingerprint: every leaf value as a path and a value, with each number reduced to the decimal it means. A value the reading dropped is still in the source and missing from the writing, and the fingerprint names it.

The fingerprint is a sorted multiset, so an xs:all whose members arrived in another order is the same document, and a repeated member that arrived twice is two entries.

Validity is the third leg, and it stays a script: xmllint against input.xsd over the written tree. A test binary that shells out to a validator would need xmllint on every machine that runs the suite.

An empty element states a value. The fingerprint work found a reading defect that the schema hides: bool in ore_types.xsd is an enumeration whose spellings include the empty string, and ORE writes <EndOfMonth/> for it. The binder marked the field present when the element opened, stored nothing when no text arrived, and left the enumeration's index at zero – which is Y, a different value. <EndOfMonth/> was being read as <EndOfMonth>Y</EndOfMonth>, and every writing validated, so nothing caught it. A leaf that closes without text now stores the empty value, which a text field keeps and an enumeration resolves to its empty spelling, and an enumeration that spells no empty string refuses the document.

6. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
Documents whose values survive the writing 246 of 246 publish/bin/ores.ore.core.tests \"[ore][xml][roundtrip-values]\" pending
Written documents that validate against input.xsd 246 of 246 ./scripts/validate_ore_examples.sh build/output/linux-clang-release-ninja/xml-roundtrip/emitted external/ore/xsd pending
XML suites 541 cases, 63,650 assertions publish/bin/ores.ore.core.tests \"[ore][xml]\" pending
Elements the model did not claim 0; none of the seven families has an xs:any grep -l 'field_kind::residual' projects/ores.ore/core/src/xml/generated/*.cpp pending

7. Notes

8. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

9. PRs

PR Title
#3449 [ore] Prove the round trip preserves the data and validates

10. Review

Comment summary File Decision Notes
       

11. Result

Merged. The harness compares two records and two value fingerprints, so a value lost on the reading is named rather than compared against itself, and it covers every document of the eight families on tables. Validity stays a script, so a machine running the suite needs no validator.

Emacs 29.3 (Org mode 9.6.15)