How the pricing engine data is sourced and curated

1. Extraction is mechanical. A person does not type what a script can read.
2. Curation is explicit. Every authored value has a reason, a source, a curator and a date.
3. Generation is deterministic. The same inputs write the same SQL, and --check fails on drift.
4. Provenance is per row, not per file. A reader can tell an extracted value from a curated one.
5. The data model owns the rules. The screen reads the valid values and the constraints; it does not restate them.
6. One enforcement point. The write path applies the rules, so every client gets the same answer.

The current seed was measured at ORE commit 3d75a6908. Recording the commit beside the data, on the DQ dataset, is part of the dataset task; a new extraction records the new commit, so a reader knows which source the rows describe.

extract_vocabulary.py reads Docs/UserGuide/pricing/pricingengines.tex and splits it on each Product Type: subsection. For each product type it records the trade type in used_by, the model and engine pairs, and for each pair the settings with their scope, name, kind, usual value, options, required flag and the guide text. The result is pricing_vocabulary.json, a derived file, not a hand-edited one.

The extraction does not decide an instrument code. It records the trade type it read, which is the input to the mapping in curation.

The guide states a setting's allowed values for some settings and omits them for the rest. The code always states the type: every engineParameter or modelParameter call is wrapped in a parser, and an enumerated parser maps the document's string to a C++ enum, so its map is the enum's own list of values.

extract_parameter_types.py reads that in one pass over the ORE source. It records, for every parameter name, the scope, the parser, the source file and line, and for an enumerated parser the strings its map lists, in the map's order. It types 226 parameters today: 50 boolean, 43 integer, 56 real, 4 period, 13 enumerated and 60 text.

The rules, all mechanical:

• parseBool gives a flag set; the setting is one of the two switch values.
• parseInteger, parseReal and parsePeriod give a set whose source kind states the type (integer, number or text) and carries no values. The screen draws the field from it and the check refuses a value that is not of the type.
• An enumerated parser, such as parsePolynomType, parseSequenceType, parseRegressorModel or parseSalvagingAlgorithmType, gives an enumerated set whose values are the parser's map keys, in order.
• parseListOfValues takes its type from the inner parser named after the list.

--merge writes curation/value-sets-code.json. The code's file is rewritten on every extraction; the guide's value-sets.json is a person's file. The seed generator composes them, the code over the guide: a code set replaces the guide's set of the same name, and a guide set no setting names after that is dropped because its link now points at the set the code states. The generator then refuses a set no setting names, an enumerated set with no values, or a link to an unknown set. Writing the code's file whole is what keeps a re-extraction from leaving a stale link behind.

The one statement the code cannot make is where a setting's choices come from: a parser sees only a string. So a guide set that names a catalogue keeps that statement over the code's plain text set of the same name, and the set carries no values of its own because the relation the catalogue names holds them. The sensitivity template is the case: ORE reads its setting as text, and the choices are the rows of ores_analytics_sensitivity_templates_tbl.

The extraction is build-time evidence. The rows it produces are written by the seed, and the screen and the write path read the rows. Nothing reads the parser, the code or the guide at run time.

The parser must wrap the parameter call directly. A wider window let one parser claim a parameter that a later call in the same statement reads, and the first run of this script typed ReferenceCalibrationGrid, a calibration grid, as a boolean for that reason; the trigger then refused the grid during provisioning. That failure is the gate working. A parameter a compound call reads, such as a ternary that chooses between two names, is not matched and keeps the guide's set or none; the review catches it by comparing the typed count with the parameter count.

Curation is data, not code. It lives in projects/ores.seeder/datasets/ore_pricing_engines/curation/, one file per concern, each entry keyed by the row it changes and carrying a reason, a source, a source_ref, a curator and a date:

"AmericanSwaption_XCcy": {
  "description": "American Swaption X Ccy",
  "instrument": "NONE",
  "reason": "No instrument is mapped yet. The mapping task decides it. The description is the value the seeded table carried.",
  "source": "seed",
  "source_ref": "projects/ores.sql/populate/analytics/analytics_pricing_engine_types_populate.sql",
  "curator": "marco",
  "date": "2026-10-25"
}

The instrument is the quality the screen groups by. It comes from the type, not from the trade, and this file is its only home. A product type the guide names but the file does not fails the generator, so the debt is visible and a person decides it.

The source is one of four categories: ore_guide for the guide, ore_code for the ORE source, authored for a text a person wrote, and seed for a value carried over from the old literals. The source_ref points at the place: the guide section, the source path, or the file the value was carried from.

The loader (curation/__init__.py) rejects an entry that lacks a reason, a source, a source reference, a curator or a date, and an entry whose source is not one of the four.

A curation entry is reviewed like code: it appears in the diff, names its source and reason, and carries a person and a date.

The target is that the ore parameter contract carries, beside the text: source (ore_guide, ore_code, authored, seed), source_ref (the guide section or the source file and line), curation_note (what a person changed), reviewed_by and reviewed_at, and that the analytics tables carry the same on the rows that describe an ORE element. A row without a source is a bug, not a default.

The DQ framework already stages and publishes the data: a dataset, an artefact table, an artefact type, a bundle and a publication, with its own provenance fields for who changed a row, when and why. The methodology uses that pipeline and adds what DQ does not hold:

• The source commit, the curation file version and a link to this page go on the DQ dataset as its metadata, next to the licence and the description.
• The coverage report (types mapped, types unmapped, settings documented, globals documented, rows authored) is written by the generator and recorded on the dataset, so a consumer reads coverage without rerunning the generator.
• The value rules (domain, enumerated values, range, required) stay in the data model and are enforced on the write path. They are not a DQ feature: DQ has no rule engine today, and its quality metrics are still a concept.

What DQ gives us for free: publication to each tenant, dataset identity and versioning, and the change provenance of the rows themselves.

Some gates are live. The parameter generator fails on a parameter without a text. The vocabulary generator fails on a product type the guide names with no curation entry, and on a model or engine with no description. An entry without provenance fails when the curation loads. The rest are the target of the provenance task. The full set fails, rather than writing a guess, when:

• a parameter has no text;
• a parameter has no value domain;
• a product type has no instrument code;
• a description is only the humanised code and the type has no curated documentation;
• a global parameter has no domain or no text;
• the same leaf would be written twice;
• a value set has no values, or no setting names it;
• a setting's usual value is not in its set;
• --check finds the committed SQL differs from the generated SQL.

ores.shell ore_parameter_settings check reads the contract and the four value set tables over the reporting subjects and prints the count with a set, the links outside the contract and any enumerated set with no values, then OK or FAIL. ore_parameter_settings list prints every parameter with its type and choices, and ore_parameter_settings show <name> prints one. The command is the live gate a person runs against an environment; the SQL view is the same facts for a query.

The coverage report prints, per run: types mapped, types unmapped, settings documented, globals documented, and the rows whose text is authored rather than extracted. A reviewer reads the report before the diff.

1. Run the extractor against the recorded ORE commit, then the type extractor with --merge and read the counts it reports.
2. Read the coverage report: every count that moved is explained.
3. Read the curation diff: every entry has a reason, a source and a source reference.
4. Spot-check one product from each asset class against the guide.
5. Run the generator and --check.
6. Run generate_value_sets.py --check and read the set and link counts.
7. Run the screens' tests; the vocabulary fixtures are built from the same data shape.

The curated facts now live in the curation files, and every generated row is unchanged by the move, which proves the files say what the literals said.

The release floor is still not curated. pricing_parameter_releases.json is measured from the ORE tags, and a row with no measured release falls back to the literal 1.8.0.7 or earlier in pricing_parameter_rows.py. A curated release file is not implemented; add one when a floor is a decision rather than a measurement.

Still the target: the provenance columns on the rows, the coverage report, the value rules on the write path, and an instrument for each product type that has none.