Task: Fix text escaping in the ORE bindings and the SQL codegen
Table of Contents
This page documents a task in the Model the ORE report configuration as entities story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Text survives import, storage and export unchanged. An ampersand, an apostrophe, a quote or an angle bracket in an ORE document is stored in the database as the character itself, and the export writes it back escaped exactly once. A string the SQL codegen writes into a literal cannot break the script it is written into.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Model the ORE report configuration as entities |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-02 |
3. Acceptance
- The ORE binding decodes entity and character references in element text, as it already does in attribute values, and leaves CDATA content untouched.
- An entity name is matched exactly:
&a;is not decoded as an apostrophe. - The fix is made in the
xsdcppgenerator and the bindings are regenerated, so the next regeneration does not undo it. - The round-trip harness compares after a full
save_data/load_datacycle again, as it did before PR #2500, and every kind passes. - After a recreate and an import of the corpus, no today's market, curve
configuration, simulation or sensitivity market row holds the text
&. - The SQL codegen escapes every value it writes into a SQL string literal. A
foreign key
error_messagewith an apostrophe regenerates into valid SQL, and a codegen test proves it. - Every existing model regenerates byte-identical after the codegen change.
4. Plan
- Reproduce at the binding: test cases that load a document with entities in element text and fail against the current binding.
- Fix the parser in
xsdcpp(src/XmlParser.cpp), with tests there. - Regenerate the bindings with the fixed generator and take the parser change.
- Restore the save and load cycle in the round-trip harness, and prove the database holds the decoded character.
- Escape the model text the SQL codegen writes into literals, with a test.
5. Measurements
Every number this task claims about a tree, with the command that produced it
and the commit it was measured at. A number without both is not a measurement,
because a clean-up record describes a tree and trunk moves under a branch: a
consumer count, a gate's component count and a suite's size all change with no
code changing, and the reader of a bare number cannot tell which tree it came
from. Where the raw output matters, put it in an evidence/ file beside the
task and name that file in the Command column.
| What | Value | Command | Commit | ||||
|---|---|---|---|---|---|---|---|
| New binding cases failing before the fix | 5 of 5 | ores.ore.core.tests "[file_io]" against the old binding |
280b15590c | ||||
| Full test suite after the fix | 89 of 89 suites pass | compass test run |
ea17fd2cd5 | ||||
| Today's market cases after the fix, harness cycle restored | 8 cases, 459 assertions pass | ores.ore.core.tests "[todaysmarket]" |
ea17fd2cd5 | ||||
| xsdcpp suites with the fix | 9 of 10 pass; Ecoa_test fails the same way without the fix |
ctest in a fresh clang build |
xsdcpp 09f0dec |
||||
| Codegen tests | 1055 pass | pytest projects/ores.codegen/tests |
ea17fd2cd5 | ||||
| Models with an apostrophe in an error message | 0 | grep -rlE "^:(book_)?error_message:.*'" projects/*/modeling |
ea17fd2cd5 | ||||
| Corpus files with an entity in element text, by kind | todaysmarket 5, curveconfig 5, simulation 3, sensimarket 1 | =grep -rlE ">[^<]*&(amp\ | lt\ | gt\ | quot\ | apos);[^<]*<" external/ore/examples –include=*.xml= | 73c00d0dfb |
6. Notes
Found while reviewing PR #2500 (today's market round trip).
The binding decodes attributes and not element text. In
projects/ores.ore/core/src/domain/domain.cpp, the tokenizer passes an
attribute value through unescapeString (line 429). Element text goes through
stripComments only (line 635), which removes comments and unwraps CDATA but
decodes nothing. The writer's escape_xml escapes all five characters on the
way out. So a text value with & loads as the literal &, and saves as
&amp;: it gains one level each cycle.
The database already holds the wrong value. After the today's market database
test ran in bright_faraday, the entry for one correlation held the decoded
text in its attribute column and the escaped text in its element-text column:
key_value | target ------------------------------+--------------------------------------------- EQ-RIC:.STOXX50E&EQ-RIC:.SPX | Correlation/EQ-RIC:.STOXX50E&EQ-RIC:.SPX
The harness change hid it. PR #2500 removed the save/load cycle from
make_roundtrip_kind in roundtrip_harness.hpp, because the cycle reported a
difference in five today's market files. The diagnosis was correct and the
remedy was not: the difference was this defect. The in-memory and database round
trips still pass, because the original and the rebuilt document both carry the
same undecoded value. Restore the cycle once the binding is fixed.
Other kinds are exposed too. Corpus files with an entity in element text, by
kind: todaysmarket 5, curveconfig 5, simulation 3, sensimarket 1.
unescapeString matches by prefix. It compares the entity name with
strncmp(i, *j, sequenceLen), where sequenceLen is the length of the text in
the document. &a; therefore matches apos. No corpus file is affected; fix it
in the same change.
The generator is out of band. xsdcpp is a sibling checkout, not on
PATH, and regeneration overwrites its output wholesale. See
xsdcpp for the checkout, the pin
and the regeneration steps.
The SQL codegen writes error_message into a literal unescaped.
_soft_fk_validation_node_to_dict in
projects/ores.codegen/src/codegen/org_loader.py copies the property as is,
and the templates emit it as raise exception '{{error_message}}' (domain
entity and junction create templates). The apostrophe in "today's market"
aborted a recreate with two of five tables built. Every consumer is a SQL
literal and no model holds an apostrophe today, so doubling each apostrophe in the loader
changes no current output. Check book_error_message on the trade model, which
reaches the template by another path, and any other model property emitted
into a literal.
Once fixed, remove the "keep an apostrophe out of error_message" step from
doc/recipes/codegen/how_do_i_add_a_new_codegen_entity_model.org.
7. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
8. PRs
| PR | Title |
|---|---|
| #2505 | [ore] Decode entity references in element text and escape SQL literals |
9. Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | Local review: the book_error_message escaping had no test |
test_org_loader_foreign_keys.py |
Accepted | Added a case for the party-from-book section. |
| 2 | Local review: the five new binding cases did not log, as the checklist and the file's other cases do | xml_file_io_tests.cpp |
Accepted | One BOOST_LOG_SEV(lg, info) per case. |
| 3 | Local review: � decodes to an embedded NUL |
domain.cpp (xsdcpp) |
Declined | Attributes already did this before the change. XML forbids the reference, and no corpus file uses it. |
| 4 | Local review: an unterminated comment in element text makes decodeText scan past its range for - |
domain.cpp (xsdcpp) |
Declined | Present before this change in stripComments, for malformed input only. Out of scope for an escaping fix. |
| 5 | PR review: holds_an_ampersand fails without saying it depends on the fixture |
todays_market_database_roundtrip_tests.cpp |
Accepted | An INFO names the fixture and what it must contain. |
| 6 | PR review: escaping in the loader means the model holds the escaped text; a template-level filter would be more robust | org_loader.py |
Declined | The templates are Mustache, which has no filters. The loader is the one place these values enter, both are used only inside SQL literals, and the helper's docstring states that contract. |
| 7 | PR review: a reference split by a comment is not decoded | domain.cpp (xsdcpp) |
No change | The reviewer advises leaving it: very rare, and identical to the generated parser. |
10. Result
Text now survives import, storage and export unchanged.
The binding. The defect was in xsdcpp's embedded parser, not in ORE
Studio code. stripComments, the element-text path, removed comments and
unwrapped CDATA but decoded nothing, while attributes went through
unescapeString. It is now decodeText, which decodes references outside
CDATA. unescapeString also matched entity names by prefix, so &a; and &;
both decoded as an apostrophe; a name must now match exactly.
The fix lives in xsdcpp (src/XmlParser.cpp and test/XmlParser_test.cpp),
so the next regeneration keeps it. The ORE Studio bindings take only the three
parser hunks of that regeneration, which are byte-identical to the generated
ones. The same regeneration also changed the trade and componentSubTrade
types, an unexplained generator drift unrelated to escaping, which would make
the trade parser accept unknown child elements. That is recorded on
xsdcpp and left out.
The harness. The round-trip harness compares after a save and load cycle again, as before PR #2500. The today's market corpus passes with it, and the database test proves an ampersand is stored as the character.
The SQL codegen. The loader doubles apostrophes in error_message and
book_error_message, the only model text written into a SQL literal. No model
holds one today, so no generated file changed. The recipe no longer tells
authors to avoid apostrophes.
Component audit. The task touched ores.ore and ores.codegen. The
automated half, projects/ores.codegen/validate_docs.sh, passes for all 41
components. The task added or moved no file, folder, type or component: it
changed one parser function, the harness comparison, one loader function and
their tests. So the shape, morphology, naming, registry and diagram checks
have nothing new to judge. The comment checks were applied to the changed code.
In xsdcpp. Committed on fix-abstract-element-refs as 09f0dec, after
2f7dc07, which commits the resolver fix in Reader.cpp that had been
uncommitted since the 2026-09-26 regeneration.