Task: Fix text escaping in the ORE bindings and the SQL codegen

Table of Contents

This page documents a task in the Model the ORE report configuration as entities story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

Text survives import, storage and export unchanged. An ampersand, an apostrophe, a quote or an angle bracket in an ORE document is stored in the database as the character itself, and the export writes it back escaped exactly once. A string the SQL codegen writes into a literal cannot break the script it is written into.

2. Status

Field Value
State DONE
Parent story Model the ORE report configuration as entities
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-02

3. Acceptance

  • The ORE binding decodes entity and character references in element text, as it already does in attribute values, and leaves CDATA content untouched.
  • An entity name is matched exactly: &a; is not decoded as an apostrophe.
  • The fix is made in the xsdcpp generator and the bindings are regenerated, so the next regeneration does not undo it.
  • The round-trip harness compares after a full save_data / load_data cycle again, as it did before PR #2500, and every kind passes.
  • After a recreate and an import of the corpus, no today's market, curve configuration, simulation or sensitivity market row holds the text &.
  • The SQL codegen escapes every value it writes into a SQL string literal. A foreign key error_message with an apostrophe regenerates into valid SQL, and a codegen test proves it.
  • Every existing model regenerates byte-identical after the codegen change.

4. Plan

  1. Reproduce at the binding: test cases that load a document with entities in element text and fail against the current binding.
  2. Fix the parser in xsdcpp (src/XmlParser.cpp), with tests there.
  3. Regenerate the bindings with the fixed generator and take the parser change.
  4. Restore the save and load cycle in the round-trip harness, and prove the database holds the decoded character.
  5. Escape the model text the SQL codegen writes into literals, with a test.

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
New binding cases failing before the fix 5 of 5 ores.ore.core.tests "[file_io]" against the old binding 280b15590c
Full test suite after the fix 89 of 89 suites pass compass test run ea17fd2cd5
Today's market cases after the fix, harness cycle restored 8 cases, 459 assertions pass ores.ore.core.tests "[todaysmarket]" ea17fd2cd5
xsdcpp suites with the fix 9 of 10 pass; Ecoa_test fails the same way without the fix ctest in a fresh clang build xsdcpp 09f0dec
Codegen tests 1055 pass pytest projects/ores.codegen/tests ea17fd2cd5
Models with an apostrophe in an error message 0 grep -rlE "^:(book_)?error_message:.*'" projects/*/modeling ea17fd2cd5
Corpus files with an entity in element text, by kind todaysmarket 5, curveconfig 5, simulation 3, sensimarket 1 =grep -rlE ">[^<]*&(amp\ lt\ gt\ quot\ apos);[^<]*<" external/ore/examples –include=*.xml= 73c00d0dfb

6. Notes

Found while reviewing PR #2500 (today's market round trip).

The binding decodes attributes and not element text. In projects/ores.ore/core/src/domain/domain.cpp, the tokenizer passes an attribute value through unescapeString (line 429). Element text goes through stripComments only (line 635), which removes comments and unwraps CDATA but decodes nothing. The writer's escape_xml escapes all five characters on the way out. So a text value with &amp; loads as the literal &amp;, and saves as &amp;amp;: it gains one level each cycle.

The database already holds the wrong value. After the today's market database test ran in bright_faraday, the entry for one correlation held the decoded text in its attribute column and the escaped text in its element-text column:

 key_value                    | target
------------------------------+---------------------------------------------
 EQ-RIC:.STOXX50E&EQ-RIC:.SPX | Correlation/EQ-RIC:.STOXX50E&amp;EQ-RIC:.SPX

The harness change hid it. PR #2500 removed the save/load cycle from make_roundtrip_kind in roundtrip_harness.hpp, because the cycle reported a difference in five today's market files. The diagnosis was correct and the remedy was not: the difference was this defect. The in-memory and database round trips still pass, because the original and the rebuilt document both carry the same undecoded value. Restore the cycle once the binding is fixed.

Other kinds are exposed too. Corpus files with an entity in element text, by kind: todaysmarket 5, curveconfig 5, simulation 3, sensimarket 1.

unescapeString matches by prefix. It compares the entity name with strncmp(i, *j, sequenceLen), where sequenceLen is the length of the text in the document. &a; therefore matches apos. No corpus file is affected; fix it in the same change.

The generator is out of band. xsdcpp is a sibling checkout, not on PATH, and regeneration overwrites its output wholesale. See xsdcpp for the checkout, the pin and the regeneration steps.

The SQL codegen writes error_message into a literal unescaped. _soft_fk_validation_node_to_dict in projects/ores.codegen/src/codegen/org_loader.py copies the property as is, and the templates emit it as raise exception '{{error_message}}' (domain entity and junction create templates). The apostrophe in "today's market" aborted a recreate with two of five tables built. Every consumer is a SQL literal and no model holds an apostrophe today, so doubling each apostrophe in the loader changes no current output. Check book_error_message on the trade model, which reaches the template by another path, and any other model property emitted into a literal.

Once fixed, remove the "keep an apostrophe out of error_message" step from doc/recipes/codegen/how_do_i_add_a_new_codegen_entity_model.org.

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

8. PRs

PR Title
#2505 [ore] Decode entity references in element text and escape SQL literals

9. Review

# Comment summary File Decision Notes
1 Local review: the book_error_message escaping had no test test_org_loader_foreign_keys.py Accepted Added a case for the party-from-book section.
2 Local review: the five new binding cases did not log, as the checklist and the file's other cases do xml_file_io_tests.cpp Accepted One BOOST_LOG_SEV(lg, info) per case.
3 Local review: &#0; decodes to an embedded NUL domain.cpp (xsdcpp) Declined Attributes already did this before the change. XML forbids the reference, and no corpus file uses it.
4 Local review: an unterminated comment in element text makes decodeText scan past its range for - domain.cpp (xsdcpp) Declined Present before this change in stripComments, for malformed input only. Out of scope for an escaping fix.
5 PR review: holds_an_ampersand fails without saying it depends on the fixture todays_market_database_roundtrip_tests.cpp Accepted An INFO names the fixture and what it must contain.
6 PR review: escaping in the loader means the model holds the escaped text; a template-level filter would be more robust org_loader.py Declined The templates are Mustache, which has no filters. The loader is the one place these values enter, both are used only inside SQL literals, and the helper's docstring states that contract.
7 PR review: a reference split by a comment is not decoded domain.cpp (xsdcpp) No change The reviewer advises leaving it: very rare, and identical to the generated parser.

10. Result

Text now survives import, storage and export unchanged.

The binding. The defect was in xsdcpp's embedded parser, not in ORE Studio code. stripComments, the element-text path, removed comments and unwrapped CDATA but decoded nothing, while attributes went through unescapeString. It is now decodeText, which decodes references outside CDATA. unescapeString also matched entity names by prefix, so &a; and &; both decoded as an apostrophe; a name must now match exactly.

The fix lives in xsdcpp (src/XmlParser.cpp and test/XmlParser_test.cpp), so the next regeneration keeps it. The ORE Studio bindings take only the three parser hunks of that regeneration, which are byte-identical to the generated ones. The same regeneration also changed the trade and componentSubTrade types, an unexplained generator drift unrelated to escaping, which would make the trade parser accept unknown child elements. That is recorded on xsdcpp and left out.

The harness. The round-trip harness compares after a save and load cycle again, as before PR #2500. The today's market corpus passes with it, and the database test proves an ampersand is stored as the character.

The SQL codegen. The loader doubles apostrophes in error_message and book_error_message, the only model text written into a SQL literal. No model holds one today, so no generated file changed. The recipe no longer tells authors to avoid apostrophes.

Component audit. The task touched ores.ore and ores.codegen. The automated half, projects/ores.codegen/validate_docs.sh, passes for all 41 components. The task added or moved no file, folder, type or component: it changed one parser function, the harness comparison, one loader function and their tests. So the shape, morphology, naming, registry and diagram checks have nothing new to judge. The comment checks were applied to the changed code.

In xsdcpp. Committed on fix-abstract-element-refs as 09f0dec, after 2f7dc07, which commits the resolver fix in Reader.cpp that had been uncommitted since the 2026-09-26 regeneration.

Emacs 29.3 (Org mode 9.6.15)