Task: Carry one canonical form for each fact

evidence/alias_copies.py reads the other two copies, and alias_copies.txt holds what it printed. The reader's tables are one statement; the refdata entities convert with a table of their own, and the conventions seeder normalises with a third before it writes a row:

business_day_conventions: 26 renderings, the reader
                         27 renderings, refdata (and 'Nearest' is not a rendering the reader accepts)
day_counters:            72 renderings, the reader
                         34 renderings, the seeder (5 of its keys no reader accepts, 43 reader rows it lacks)
frequencies:             18 renderings in both

So the story's claim is measured rather than asserted: three tables in three components hold the same knowledge, and two of them disagree with the reader.

Comparing the reader's tables with the parser map that accepts the renderings found two defects, both fixed in this slice:

The check that found them now runs on every build: a rendering one source names and the other does not stops the generator and names the rendering.

ores_reporting_ore_parameter_value_set_values_tbl gains canonical_value, null when the row is the canonical form and the canonical value when it is another spelling of one. The set's values therefore state both directions, and a screen asks for the rows whose canonical_value is null.

The alternative, a second table of aliases, states the same fact twice: the alias row would carry a value and a pointer to a value row, and the set's own value would be the only thing that decides what is canonical. One column keeps the row count and the ordering the set already states.

convention.business_day_convention, convention.day_counter and their two curve. twins were catalogue sets: their values were the rows of the refdata registers, and a catalogue set holds no value rows, so it has nowhere to put a canonical form. The registers hold every rendering ORE accepts — 72 day counters, where 23 facts exist — because a guard refuses a spelling the register lacks. A set drawn from them therefore offered the same day counter four times, which is the defect the story is about, in the one place a person sees it.

So the four sets state their renderings themselves, and the registers keep their own job: they are the whitelist a column guard checks, and the rows a convention holds. A rendering is a fact about a parameter, so it lives with the parameter's rule, not with the entity that happens to spell it. The registers still accept every rendering, so no stored value changes and no guard loosens.

The alternative, marking the rendering rows in the registers, would have made a 26-row convention register hold 19 spellings, and every screen that lists a register would have to filter rows that are not conventions.

The mappers convert today, before the store sees a value, so the store's guards never meet a spelling. That is why the knowledge had to live in code: a value that never reaches the database cannot be converted there.

A helper, ores_reporting_canonical_value_fn(set_code, value), returns the canonical form of a value the set knows, and the four leaf guards call it as they walk the column they are about to write: NEW.column = ores_reporting_canonical_value_fn(...). The guard is the one boundary every writer passes through, so the shell, a screen, an import job and a publish function all convert once, in one place.

The columns are guarded but not rewritten today, and a BEFORE trigger may rewrite NEW, so no schema change is needed to carry this.

The change has a state that is worse than either end: mappers stopped converting and the guard not converting yet would let a spelling reach a row unwatched. So:

1. Seed the renderings into the sets from the compiled tables, with a generator that reads conventions_fields.cpp, so the set states what the table states and a check compares them. Nothing changes at runtime, and a combo shows a fact once for the first time. Done, PR #4244.
2. Teach the guards to convert, and keep the mappers converting: the guard's conversion is then exercised by a test that writes a spelling straight to a table, and the import path still converts first, so no row changes. Done, PR #4248.
3. Stop the mappers converting, in one commit, so the spelling reaches the guard for the first time. The round-trip suites prove nothing was lost, and a check that no table holds a non-canonical value proves the guard converted.
4. Delete the compiled tables and the census script's reason to exist, which is the story's "no compiled table holds ORE's spellings".

Slice 1 is data and a generator; 2 and 3 are the runtime change; 4 is the deletion that makes the claim true.

A query over the map and the guarded tables: for every child row, the value a row holds must equal the canonical form of its set, so a spelling that reached a row by any path is named with its table, its column and its set. The check reads the map, so a column that gains a guard gains the check with it.

A value row names the form a configuration holds and is empty when the row is that form, so one table states which renderings are accepted, what each means and which one a configuration carries. The four sets that drew on a refdata catalogue became enumerated, because a catalogue set holds no value rows and so had nowhere to put the column; the registers keep the whitelist a guard checks.

The numbers, measured after the seed and read back through ores.shell:

The renderings are read from ORE's reader and its parser rather than curated, so the check that they agree runs on every build. It found five defects, all fixed: three renderings the reader refused and five rows it could never apply.

Acceptance, as it stands:

• Every parameter whose ORE reader accepts more than one rendering is named, with the reader that accepts it and the canonical form chosen for it. Done, by ore_aliases.py reading the reader's six tables and the five parsers.
• The map from an accepted rendering to the canonical form is data, and no compiled table holds ORE's spellings. Half done: the reading direction is data; the direction a write emits is still day_counter_spellings and bdc_spelling, because the store's form is not always a form a document may write. Slice 4 states that direction as data too.
• A write converts through one boundary and refuses a rendering the map does not know, naming the set. Done: the four guards convert, and a value outside the set is refused with the set's name.
• The screens offer the canonical forms and the guard accepts every rendering. Done: a combo offers the canonical forms, and the guard accepts a rendering and stores its canonical form.
• A check fails when a table holds more than one rendering of one fact. Not yet: slice 3's query over the map and the guarded tables.

ores_reporting_canonical_value_fn(set_code, value) answers with the form the set names for one value, and with the value itself when the set states no other rendering of it. All four guards call it after they accept a value, so a rendering does not reach a row by any of the four write paths.

Two of the guards walk columns they read from a table, so they rewrite the row through its JSON form: a trigger cannot name a column that a query named.

The measurement that matters, on a freshly recreated database whose conventions were published afterwards:

$ psql -f tmp/noncanon.sql
NOTICE:  checked 73 columns: 0 hold a rendering

The 73 are the guarded columns whose set states a rendering. Before this slice the same query named 45 rows, all of them A365, a rendering of ACT/365.FIXED, written by the conventions publish: the publisher is a writer that never went through the mappers. After it, the same publish leaves ACT/365.FIXED in all 45.

ores_reporting_canonical_value_violations_fn() names every live row in a guarded column whose value is not the form its set names, with the table, the column and the set. It reads the element map, so a column that gains a guard gains the check with it, and it reads only the live version of a row: a value written before the rule existed is history, not a defect.

On the database whose conventions were published after slice 2 it answers nothing. Run it with:

./compass.sh sql -- -c "select * from ores_reporting_canonical_value_violations_fn()"

Stopping the mappers converting was tried, and the measurement stopped it. The reader was changed to check a spelling against its table and hand the spelling on, which is the shape the task describes; ores.ore.core.tests then failed on 72 pinned exports of the ORE corpus, because that suite models the row path without a database, and the canonicalisation it used to perform in memory now happens in the guard.

The suite's pinned bytes therefore describe a pipeline that has a canonicaliser. Reverting the readers leaves them describing the one that exists; keeping the change means re-pinning 72 documents whose only difference is that they keep the spelling the document wrote, and accepting that the suite then measures the codec rather than the store.

Two ends are honest and this tree can have either, not both:

The second end is where the story points, and it needs the export spelling stated as data before the codec can stop. That is the next slice: a column on the value row for the form an export writes, or a writer that reads the set. Until then the compiled tables have a reason to exist, and slice 4 waits on it.

Slice 2 made the store hold one form of each vocabulary value, and five database round-trip cases compared the export against the document that went in. They read the canonical form as a defect. They now state the round trip on what the store returns, and every count assertion stays.

The same run found a list rule that refused a good list: a value written across several lines carries the newline and tab around each item, and btrim trims spaces alone, so a multi-line list of periods was refused. The guard now trims the whitespace around an item and still refuses an item that is nothing but whitespace.

Three eventing cases in ores.refdata.core.tests fail on a synthetic day counter code (A365-0) that the guard refuses. The generator invents the code, and the guard states the set; they fail on main too and are not this task's work.

Three things came out of getting main green after these slices.

A test that seeds a register row and then names it in a config relied on the set being the register's live rows: it invented the day counter code A365-0, and the catalogue view showed it because the view holds every tenant's rows. The set now states the spellings, so the fixture states one (A365). That is the behaviour the story asked for, and the three eventing cases are the only place that relied on the old one.

Eight settings whose document value is a comma separated list were stated as one value, so the guard refused a good list. Their sets now state cardinality = many: CalibrationMoneyness, BucketTimesDeltaGamma, BucketTimesVega, Bootstrap.CalibrationGrid, ReferenceCalibrationGrid, GridCoarsening, calibrationIndexTerms, recoveryRateProbabilities, QuantoCurrencyPairs and Pricing.ConversionRatioDiscretisationGrid. The list rule already trims the whitespace around an item, which a multi-line element carries.

Heston.FixedValues is the tenth, and it needs a set of its own: it is five flags, and it links to the shared flag set, which cannot become many without turning every boolean parameter into a list. That is curation work, not a schema change, and it is recorded here rather than guessed at.

The store's form and a document's spelling were two different things for four facts: ACT/360 (incl. last), ACT/365 (Canadian Bond), ACT/365 (JGB) and Nearest were forms no document could write, so the codec held a second table to spell them on the way out. Each now takes the spelling the schema states: ACT/360 (Incl Last), Act/365 (Canadian Bond), Actual/365 (JGB) and NEAREST.

With that, day_counter_spellings, bdc_spelling and both helpers are gone, and put_day_counter and put_bdc write the stored form as it stands. The generator holds the invariant: a table that makes a form no rendering writes stops the run, so the write direction cannot come back.

The export changes for the 20 forms the old table rewrote into another spelling: a document that says A365 exports ACT/365.FIXED where it used to export A365F. The 72 pinned corpus digests moved once, written by the case itself under ORES_REGEN_CONVENTIONS_DIGESTS, and the rendering counts fall from 52 and 20 to 49 and 19 because three forms became renderings of themselves.

The reader still converts: a document read has to reach a column. Deleting that table leaves the guard as the only converter, and it waits on the sets being authored rather than generated from the table.

A curve's day_counter was ruled by two things at once: the value set, which states the renderings and the form a configuration holds, and the entity's soft foreign key to ores_refdata_day_counters_tbl, which states the spellings a register happens to hold. A generated code such as A365-0 was the visible symptom: the register admitted it and the set refused it, so which authority spoke depended on the trigger order. The register also left a hole the tests hid: the guard stores ACT/365.FIXED, which a tenant's own register may not hold, so a later update of that row could fail a check the write had passed.

The sixteen entities that named the register as a foreign key no longer do, so a day counter column has one authority. The table stays as reference data – the list of day counters ORE's schema names, which a person may read – and states no rule. Which spellings a column accepts, and which form a configuration holds, is the value set's.

Slice 2 adds ores_reporting_canonical_value_fn(set_code, value) and calls it from the four leaf guards while the mappers still convert. The helper reads a set's rows: the enumerated values, or the catalogue relation the set names. It returns the value when the set holds no row for it and the guard's whitelist is what refuses it, which is the one case a reader has to keep straight.

The three copies of the knowledge named in Discovery are still there. The refdata conventions_code header and the seeder normalisers are slice 3 and 4 work: once the guards convert, each has no reason to.

Five tables in conventions_fields.cpp held 132 renderings over 55 forms, and ore_renderings.json held the same knowledge — written from them. Two copies in one direction: the file was a projection of the C++.

The direction is now the other one. The file is the statement, and ORE's reader converts with tables generated from it by projects/ores.ore/scripts/generate_renderings_tables.py into projects/ores.ore/core/src/domain/conventions_renderings.hpp. The generated header is a copy that cannot drift: --check fails when the committed header differs from the file, and when conventions_fields.cpp states a code_table of its own.

That second half is the point. A generated copy is one statement that survives being compiled; a table written by hand next to it is a second statement, and the check now refuses one. reader_tables() and the three regular expressions that read the reader's source are gone with it, and so is generate_value_renderings.py, which wrote the file from them.

Behaviour is unchanged, and the measurement is the tables: the generated header states the same 132 pairs over the same 55 forms as the tables it replaced.

$ python3 projects/ores.ore/scripts/generate_renderings_tables.py --check
55 canonical forms across 5 tables match projects/ores.sql/utility/ore_renderings.json.

When the tables were replaced, a script parsed the five out of the previous commit and compared them with the five the header states: 132 pairs each, and no pair only in one of them.

The header compiles in the unity translation unit that holds conventions_fields.cpp, which is the only file the change touches in C++.

The plan above said "delete the compiled tables". They are still compiled, because the reader must convert and a reader cannot read a JSON file at run time. What changed is that they are no longer a second authority: nothing reads them to learn what the mapping is, and the file wins wherever the two differ.

Two pieces of the old design outlived the slices above: a codegen hook whose only user the story had deleted, and a knowledge cluster that still stated the design the hook belonged to.

The hook, store_transform, let a column name a function the mapper calls on the way into the row, plus the header that declares it. #4431 removed its one user, stored_business_day_convention, so no model set the property, and template_variables.org still named the deleted header as its example. The three template branches, the include gate in core.py, its test and both doc mentions are gone. Generated output does not change, because a branch only emits when a column sets the property.

The knowledge cluster was the record of the old design. Three pages put the conversion in the entity mapper, and the order page gave the codec as the reason it could not be anywhere else. The codec converts a conventions document on read, from the generated tables, and the guard converts every write the codec did not. The pages now state the two call chains with the lines that do each step, the two figures say the same, and the inbox capture the cluster was filed against records that both defects it names are fixed.

$ python3 build/scripts/local_checks.py --exclude-heavy   (docs branch)
✅  local checks: 9/9 passed
$ select count(*) from ores_reporting_canonical_value_violations_fn();
0

Three checks failed on this story's branches, and all three failed in a clean worktree of origin/main too. Two are fixed in #4444, because the branch could not go green until they were: the junction-events test now states the subject_prefix the template emits since the events namespace landed, and ores_schema.dq_staging.puml is regenerated. The third, component-drift, names the msgpack-c lines a337e95529 added by hand to projects/ores.storage/service/src/CMakeLists.txt; it belongs to the storage model and stays open. All three are recorded in The local check gate is red on main.