Task: Consolidate the ORE element maps

One table states which ORE element a leaf column is and which value set states its values, with the projection columns attached to it, and one link shape joins a parameter or a leaf to its set. Nothing else in the story has to guess which of three maps wins.

• One table keys a leaf by its address, (schema_name, xpath), and it is the only statement of which value set governs a leaf.
• Every projection column that carries a leaf is attached to that map, so both guards read the map and the parameter contract is no longer a prerequisite of enforcement. Done: both guards join the child and the map.
• Every row of the three tables is represented in the new map, and a test counts the leaves and the columns before and after. Done: reporting_ore_element_map_test.sql states the 480 leaves, the 482 columns, and that neither side holds a row the other does not.
• Both guards' pgTAP suites pass unchanged.
• The leaves the maps name and the contract does not describe are listed by the map, each with a decision: a value set, or text. Done: every one of the 480 names the authority that decided it, and none is left unaccounted.

Measured from the seed scripts rather than from a database, so the numbers do not depend on which schema a checkout's database holds: evidence/leaf_maps_overlap.py, with its output in leaf_maps_overlap.txt beside it.

The first cut keyed the map by (schema_name, element_name). That key is wrong, and a test found it: SpotDays is xs:integer in two conventions and xs:string that ORE reads as a period in a third, so a rule keyed by the name gave the commodity forward's integer leaf the CMS spread option's period rule, and a count of days was refused.

The key is the leaf's address in its schema. element_name stays beside it, so a person can read and search the map, and a vocabulary that holds for every leaf of a name (Currency, DayCounter, BusinessDayConvention) is stated once in ore_parameter_vocabularies.json and applied to every address that declares it. That is a rule, not a fallback: the declaration says the name is a vocabulary, and the map says which addresses declare it.

Measured with that key: 480 addresses over 482 columns and 288 element names, of which 181 addresses take text, and 31 carry a curated vocabulary.

The map is one table keyed by the ORE element, with the projection columns as a child. Both guards read it directly, so enforcement stops depending on the parameter contract, and the seeding stops depending on a column being noticed.

The map and its child are seeded, and the switches are built, but two things had to be settled before the guards could read them.

181 of the 480 addresses take the shared text set, which admits every value. They are not one problem. evidence/text_addresses.py splits them, and its output is beside it:

The 26 and the 12 are the cheap end, because a rule already exists for the same element somewhere. The 67 are the real curation: mostly lists of tenors, strikes, dates and indices, where the honest answer is often a stated decision that the leaf is a list rather than a single value.

Two examples of the 26, and why they matter: SettlementDays, FixingDays and SpotDays state text at some addresses while another address of the same name takes a rule, and DayCounter and BusinessDayConvention state text at two addresses while the map elsewhere states a catalogue. Each is either a wrong statement or a name that is two leaves, which is the SpotDays problem again.

Five of the 26 are now corrected. The extractor states no parser for those members, so it typed them text; the ORE schema states xs:integer for every declaration of SettlementDays and FixingDays and for the commodity forward's SpotDays, and convention_value_sets.py now states that type with the reason. The unconstrained count falls from 181 to 176.

Twenty-one are left, and they are not one job either. The Index, Conventions and Calendar addresses state text where a sibling states curve.text, which is the same rule under another code, so the question there is whether two text sets should exist, not whether the leaf is text. The DayCounter and BusinessDayConvention addresses at IborIndex and OvernightIndex state text where a catalogue exists.

Those three are now stated as the catalogue, and the ORE source is why. The extractor typed them text and it was faithful: IborIndexConvention::fromXML reads DayCounter and BusinessDayConvention with getChildValue into std::string and never parses them (OREData/ored/configuration/conventions.cpp:428 and :430), and OvernightIndexConvention reads its day counter the same way at :470. So the engine accepts any string there and a typo in one changes nothing, silently.

The field means the same vocabulary the other twenty conventions state, so the rule is the catalogue and the engine's permissiveness is not a licence. A value the catalogue does not hold is refused with a sentence that names the set, which is a better answer than a typo the engine ignores. The unconstrained count falls to 173.

What was left of the 26 was eight addresses of Index, Conventions and Calendar that state text where a sibling states curve.text. Those two are one rule under two codes: both are the text kind, both are named Text, and neither constrains anything. They are now one set, text, and curve.text is gone.

The collapse corrects a number this page carried. Counting only addresses that name text understated the gap, because curve.text was counted as a rule when it constrains nothing. The honest statement of the map is: 274 of the 480 addresses are free text, and 206 state a real rule. 195 of the 274 are a decision the contract makes; 79 have no statement at all.

So the 26 contradictions are closed, and the remaining work is the 79 addresses no contract row describes, plus a review of the 195 that say text where the schema declares something narrower.

The 79 were not a curation backlog. They were the signature of a wrong source.

A leaf's set was resolved through the parameter contract: the leaf's xpath joined a contract row, and the row's (scope, name) joined the setting link. The contract holds a row only for a leaf a person wrote text for, so a leaf ORE reads and the guide does not describe had no rule and took text by omission. Measured with the extractor as it stood, 79 of 480.

The rule now comes from the parser the ORE source reads the leaf with, through the curation the value set seed is written from. A leaf's rule and the rule of the setting that shares its name are one fact, so the map reads the resolved curation (generate_value_sets.resolved()) instead of deriving the answer a second way, and the contract keeps only its own job, which is documentation for the screens.

That changes what a row says. source is ore_code, source_ref is the ORE source line that reads the leaf, and reason is generated from the parser and the set's kind, so a reviewer checks the rule instead of trusting it. The 232 leaves that stay text state why they stay text: ORE never parses the value, or it does and this tree holds no catalogue of the names it accepts.

Three numbers moved, and only one of them by curation:

The 14 new rules are leaves the extractor already read and the contract did not describe: BMAPaymentConvention, BMAPaymentLag, OvernightLockouDays, Accuracy, Guess, MaxEvaluations, the two NthWeekday/Weekday leaves, BalanceOfTheMonth, OffPeakHours, Deposit/SettlementDays, LongTermAtmType, LongTermDeltaType and UseLastFixingDate. The 51 that fell out of the "no contract row" group did so because they now have a term: 37 of them state text with a reason, and 14 state a rule.

The 28 that remain are the extractor's gap, not the model's: extract_curve_terms.py reads getChildValue, getChildNode, getNodeValue and their typed forms, and these leaves are read with getChildrenValuesAsStrings, getChildrenValuesAsDoublesCompact or a name built from a variable. UNREAD_ALLOWED in the generator holds the number so it can fall and not grow.

The 28 leaves the map could not rule were mostly lists, and the reason was the extractor. `READ` knew `getChildValue`, `getChildNode`, `getNodeValue` and their typed forms; ORE reads a list with `getChildrenValuesAsStrings`, `getChildrenValuesAsDoublesCompact`, `getChildrenValuesAsPeriods` and `getChildValueAsPeriod`, and none of them matched. A leaf read that way produced no term, so no parser stated a rule and the leaf took text.

Two changes, both in extract_curve_terms.py:

• the four accessors are read, and each records the type of one item: a template that names its item (parseListOfValues<Real>) states a list of numbers, a template that names none states a list of strings, and the compact forms state what their name says;
• a leaf read with getNodeValue takes the type of the call that wraps the read, so parseListOfValues<Real>(getNodeValue(node)) is a list of numbers and parseBool(getNodeValue(node)) is a flag.

The second change is why the count moved so far. A report grid's leaves were all typed text because the read is a getNodeValue, and the parse sits around it; reading the wrapper types the grid's flags as flags and its expiry and strike lists as lists. The parser reuses one variable for every child, so a read is matched to the child that precedes it, bounded by the next read into the same variable: without the bound the first child's parser typed every later child and a grid's strikes became the flag beside them.

The 7 that remain: SwapTenors, BondTenors and SmileSwapTenors are named by a string the code builds (underlyingLabel_ + "Tenors"), so no read spells them; UseEffeciveVolatility is spelled that way in ORE and read elsewhere; DefaultCurve/Type, IndexInterpolated and Qualifier the extractor still does not reach.

A value set stated a kind and nothing else, so it could say what one period is and could not say whether the document writes one period or a list of them. ORE writes a list into a single element, and a numeric set already split on commas to admit one, with a comment saying why. That was a rule in the wrong place: it applied to every numeric parameter, whether or not its element holds a list.

ores_reporting_ore_parameter_value_sets_tbl gains cardinality, one or many. The kind states what one value may be; the cardinality states how many. ores_reporting_value_in_set_fn splits on the comma when the set says many and holds every item to the kind, and refuses an empty item, which is a trailing or doubled comma. A set of one refuses the comma, because a comma there is a document the engine cannot parse. The numeric split left the function with the comment, because curve.number_list now says the same rule as data.

Keeping the two facts apart keeps the kinds at nine. The alternative is a kind per combination, period_list beside period and so on, and then a screen reads a name to know whether its control repeats; with a cardinality it reads the kind for the control and the cardinality for whether it repeats.

The parser says what the engine accepts; it is not the only authority. Where the parser reads a leaf as a plain string, the schema still declares a type, and a leaf declared xs:integer is not free text: a document that writes two there is one the engine cannot parse, even though the parser hands the string on.

ore_schema_types.py walks the schema from the document's root element to the leaf, following a named complex type and an inline one, and states the leaf's type when that type is narrower than xs:string. A type the schema enumerates is left alone for now, because the values are the schema's and the set needs labels of its own.

The schema speaks in two places. A leaf the parser reads and states no more of takes the schema's type, after the parser's own answer. A leaf the extractor cannot reach at all takes the schema's type too, because nothing else speaks, and the row says so. The parser stays first in both, so a leaf the parser reads as a number keeps the number rule even where the schema is looser.

evidence/schema_types.py prints the two authorities side by side. Before this slice 16 leaves stated text where the schema declared a narrower type: six xs:integer, four xs:nonNegativeInteger, five bool and one xs:double. Now none does.

The 5 that remain are the three tenor leaves a variable names, DefaultCurve/Type and Qualifier. The schema types none of the five, so text is the honest answer rather than a gap. The 13 leaves whose schema type the reader does not resolve are listed by the evidence report; they all state a rule from another authority.

A set of cardinality many stated how many values a leaf holds and not what one may be, so 16 leaves said "a list of anything". The ORE user guide states the item for most of them in the sentence it describes the leaf with: Tenors and AtmTenors are "a comma separated list of valid tenor strings", Terms is "a comma separated list of tenors", SmileOptionTenors are "option expiries, in period form", CapStrikes and FloorStrikes are the strikes, DetachmentPoints are "any positive number less than one", and SmileSpreads are strike spreads.

curve_value_sets.py now states that per name, and each entry carries the reason and the guide or code line it came from, so the map's row for a tenor list reads ore_guide and cites the line. Twelve leaves gained an item rule: eight take curve.period_list and four curve.number_list.

Four keep curve.list, and their reason says why rather than leaving the gap unstated. A delta is written 25D, which is neither a number nor a period; an expiry may be a tenor, a date or a wildcard *, which the guide states; and a pillar date is a date, for which this tree has no kind yet. A date set is the missing piece, and it is the next one.

The seeded corpus holds real tenor and strike lists, and it publishes under the new rules unchanged, which is what proves the item rule is not too narrow.

An enumerated simple type in the schema names its values, so the rule for a leaf of that type is a set of them and not a shape one value must have. The schema states 142 such types, and 33 mapped leaves had text where one of them applied: Dimension with 4 values, VolatilityType with 3, Extrapolation with 4, the commodity future weekday with 7, the default curve's Type with 8, and the segment types of a yield curve.

ore_schema_types.py now reads them, and curve_value_sets.py and convention_value_sets.py generate one enumerated set per type, named curve.<type> or convention.<type>, with the values in the schema's order and the file and line the schema states them at.

Two rules keep the sets honest. A set the schema states is generated only for a leaf a column carries, because a set nothing names and no row carries is dead data; that is what the curve value-set test's orphan assertion catches. And the schema lists the empty string where a leaf may be left unset, which the guard treats as absent rather than as a choice, so the set states the choices and the unset case stays an empty field.

The scheme also closed one of the five untyped leaves: DefaultCurve/Type is declared defaultCurveType in the schema, which enumerates eight values, so the map states them without any ORE source reading the leaf.

Four leaves were still unread, and all four were the same shape: ORE names the element from a label the class passes to its base constructor, so no read spells the name. The swaption class passes ("Swap", "SwaptionVolatility", "SWAPTION", "") and the yield class ("Bond", "YieldVolatility", "BOND_OPTION", "Qualifier"), and GenericYieldVolatilityCurveConfig::fromXML reads underlyingLabel_ + "Tenors" and qualifierLabel_.

extract_curve_terms.py now reads those two forms and expands them once the class's labels are known: one term per label, named as the document spells it. Swaptions gain SwapTenors and SmileSwapTenors, yield curves gain BondTenors, SmileBondTenors and Qualifier, and the empty qualifier label states no leaf, because the swaption class never reads one.

The three tenor leaves take curve.period_list, citing the guide where it states the leaf in period form and the class constructor where it does not. Qualifier states text with the reason the parser gives: the engine reads it and never parses it.

UNREAD_ALLOWED is now zero, so the check fails the moment a projection column carries a leaf no authority can state a rule for. Coverage is complete in the sense the task set out to prove: every leaf the map carries names the authority that decided it.

Three leaves take their rule from the ORE schema because no ORE source read names them at all: UseEffeciveVolatility, which ORE spells that way and reads elsewhere, DefaultCurve/Type, whose projection path the schema types but no read matches, and IndexInterpolated. Their rows say the schema is the only authority that speaks, so the difference between "the parser read it" and "the schema typed it" is read rather than assumed.

A schema simple type states more than an enumeration. It restricts its base to a pattern, or to a range of values, and both are rules the model can hold:

• date states the pattern ORE's date parser accepts, and its own comment says it "is not super strict, e.g. 2016-17-39 would pass, but it catches most errors";
• calendar states a pattern naming every calendar ORE resolves, including the joint form US,MWK that ORE writes by repeating the group;
• dayOfMonth restricts xs:integer to 1 through 31.

So ore_schema_types.py now reads the restriction facets: base, pattern, minInclusive and maxInclusive. A built-in type keeps the family's static set, because a flag is a flag and the static set accepts the Y and N that ORE also writes; a named type the family has no static set for, or one that narrows a kind with bounds, generates a set of its own.

Sixteen leaves gained a rule: the thirteen calendars the projection carries, the base correlation's StartDate, and the two commodity future day-of-month leaves. PillarDates gained an item rule too, curve.date_list, so a list of dates is stated rather than left as a list of anything.

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

The map is one table keyed by a leaf's address, its child names the projection column that carries the leaf, and both leaf guards join the pair, so the parameter contract is no longer a prerequisite of enforcement.

Every one of the 480 leaves states a rule or the reason it has none, and each row names the authority that decided it: the ORE parser for 207 rules and 171 text decisions, the ORE user guide for 14, and the ORE schema for 88. No leaf is left unaccounted, and UNREAD_ALLOWED is zero so one cannot arrive quietly.

The value set states what one value may be by its kind, how many the document writes by its cardinality, and the values, pattern, bounds or catalogue it draws on. A set is generated from the parser's map or the schema's restriction wherever a source states it, and curated with its reason where a person does.

What the work found that the plan did not expect: the contract was the wrong source for a leaf's rule, a set could not say how many values it governed, the extractor could not read a list or a name a class builds, and the schema was an authority nobody consulted. Each is a case of a rule that lived in prose, in a name, or in another table rather than in the model.

What is left belongs to the next task. A setting link is keyed by (scope, name) and so cannot state two rules for two leaves of one name: the equity volatility curve's Expiries is a list of dates and the report grid beside it is a list of periods, and only the drawn leaf is linked. No drawn leaf disagrees today, so no data is wrong; the key is what has to change before a screen can draw both.