Task: Make the parameter data model the source of valid values and rules

Table of Contents

This page documents a task in the Make every configuration screen follow one standard story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

One parameter model states a parameter's control and its rule, and every screen reads it from there. A parameter's allowed values, its bounds, its pattern and its catalogue live in the value set tables and nowhere else. The per-pair kind and options columns go, because the value set now says the same thing once instead of on every pair that takes the setting. The engine run settings reach the same contract, so the Setup parameters stop carrying a second copy of their control in the group settings table. The screens keep one source, the write path keeps one enforcement point, and a build check fails when a parameter loses its set.

2. Status

Field Value
State DONE
Parent story Make every configuration screen follow one standard
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-30

3. Acceptance

  • The value set tables are the only place that states a parameter's control and its rule. No screen, server or seed restates an allowed value.
  • The 15 settings whose set said text although the guide lists their choices state an enumerated or boolean set, from a curation entry that names the guide as its source. The settings that stay text say in the set why they are text.
  • ores_analytics_pricing_pair_settings_tbl holds no kind and no options column. The C++, the protocol, the SQL, the TypeScript, the shell recipe and the seed are generated from the model without them.
  • usual_value stays, because the pricing screen shows the usual setting of a pair, and it states a value and not a rule.
  • Every Setup parameter that the engine run settings screen draws has a value set and a link, and that screen draws its control and checks its value from the set through the same fields the pricing screen reads.
  • A check in build/scripts/local_checks.py fails when a parameter a screen draws has no value set, when a set breaks its own rule, or when a per-pair override column returns.
  • The build, the four test suites, the drift checks and a live environment prove the behaviour: a change to a set changes the screen's choices, a refused value names its set, and the ORE export and import still roundtrip.

4. Plan

4.1. The inventory, taken before any change

The inventory was taken at commit 92fd6990bc85. Counts came from the curation and the seed files, which are deterministic, and from the restored database, whose schema is a4924adbc941bfc7 and is therefore a cross-check only; the seed row count is stated beside each database count.

4.1.1. The two tables

  1. The per-pair settings table

    ores_analytics_pricing_pair_settings_tbl (model ores.analytics.pricing_pair_setting, create analytics_pricing_pair_settings_create.sql) holds one row per (type, model, engine, scope, name): 834 rows and 126 distinct (scope, name) settings. The columns that state a value rule are:

    column holds
    kind how the screen draws the value: select, number, tenor, reference or text
    options the allowed values of a select, as one comma separated string
    usual_value the value ORE's examples use
    is_required whether the screens start a new product with the setting

    The kind counts are number 332, select 291, reference 140, text 67 and tenor 4. 291 of the 834 rows carry a non-empty options, which repeats the same list on every pair that takes the setting.

  2. The value set model

    The newer source is four reporting tables, read through the view ores_reporting_ore_parameter_settings_v (create reporting_ore_parameter_settings_view_create.sql):

    table holds rows
    ores_reporting_ore_parameter_value_sets_tbl the set: its code, source_kind, catalogue_code, minimum, maximum, pattern 218
    ores_reporting_ore_parameter_value_set_values_tbl the values of an enumerated set, with a label and an order 202
    ores_reporting_ore_parameter_value_set_catalogues_tbl the relations a catalogue set may read 5
    ores_reporting_ore_parameter_setting_value_sets_tbl the link from (scope, name) to its set 712

    The source kinds are text 72, number 63, integer 42, enumerated 29, catalogue 7, boolean 3 and regex 2. The one enforcement point is a trigger on the analytics parameter table, analytics_pricing_model_product_parameters_value_set_trigger_create.sql, which calls ores_reporting_validate_parameter_value_fn. It reads the set by (scope, name) and reads no per-pair column.

4.1.2. The consumers

  • The BFF vocabulary. vocabulary() in pricing-engines.ts reads the pair settings and serves, for each setting, kind from the row and options from the row's comma separated string, beside the value set fields valueSet, sourceKind, choices, minimum, maximum and pattern, which it also serves as the valueSets map keyed (scope, name). The reader is ore-value-sets.ts, which registers 8 catalogue relations.
  • The web pricing screens. pricingEngines.ts parses kind and options into the vocabulary type. parts.tsx reads setting.kind to decide one thing: when it is reference it draws the sensitivity template's asset class and method. The control itself already comes from controlOfValueSet() in control.ts, which reads the value set alone.
  • The analytics parameter rows. ores_analytics_pricing_model_product_parameters_tbl holds the values a product carries and is guarded by the trigger above.
  • The seeder curation and its loaders. projects/ores.seeder/datasets/ore_pricing_engines/curation/ holds value-sets.json (the guide's sets), value-sets-code.json (the sets the type extractor writes), value-sets-review.json (a person's settlements), engine-settings-groups.json (the Setup groups) and settings-code.json. curation/__init__.py loads them and refuses an entry without a reason, a source, a source reference, a curator or a date.
  • The generators. generate_vocabulary_sql.py writes analytics_pricing_pair_settings_populate.sql, including kind and options. projects/ores.sql/utility/generate_value_sets.py writes reporting_ore_parameter_value_sets_populate.sql and fails when a contract parameter has no set. generate_engine_settings_groups.py writes the two Setup tables from engine-settings-groups.json. pricing_parameter_rows.py writes the ore parameter contract, and renders the guide's options into the format prose as One of: A, B, C.
  • The codegen. The analytics entity model renders the pair setting into C++, SQL, the wire protocol, the TypeScript domain and protocol, the shell commands and the shell recipe document. The model's Table display, Columns and Detail fields state kind and options, so every one of those carries them.
  • The shell. pricing_pair_settings list prints the row, kind included. ore_parameter_settings list|show|check already reads the sets and the links, and is the live gate.

4.1.3. Where kind and options are read today, and what depends on them

reader reads depends on it for
BFF vocabulary() kind, options the reply's kind and options fields
web parts.tsx SettingValue kind the sensitivity template heading when the kind is reference
web api/pricingEngines.ts kind, options parsing the reply
generated C++, protocol, TypeScript, shell, recipe kind, options the row's fields
generate_vocabulary_sql.py the guide's kind, options the seed's two columns

No validator reads them. The trigger reads the sets. The ORE XML export and import in projects/ores.ore read neither: the options in that component are instrument options and have nothing to do with this table, and the import writes values through the save that the trigger guards. is_required is read by the screen and is not a value rule, so it stays.

4.1.4. Which parameters still have no value set, and why

  • The pricing, convention, curve and global parameters have no gap: 251 parameters (61 ModelParameters, 173 EngineParameters, 17 GlobalParameters) carry 1 link each, and generate_value_sets.py fails on the first one that does not. The links also cover 461 convention and curve leaves, which is why 712 links stand against 251 pricing parameters.
  • 72 sets are text. That is the deliberate statement that a parameter is free text, and it is data, not a gap. It is also under-stated for 15 of them: the guide lists their choices and the code typed them as text, and a code set replaces a guide set. They are Curve, FepCurve, OptionExpiryPosition, Policy, RegressorModel, SensitivityDecomposition, SeparateCallPutVols, SpreadOnIncomeCurve, useIndexCDSPricer, Calibration, CalibrationStrategy, FxCalibration, ReversionType, TreatSecuritySpreadAsCreditSpread and VolatilityType.
  • The Setup scope is the real gap: the engine run settings screen draws 15 settings and none of them has a set or a link. Their control lives in ores_reporting_engine_settings_group_settings_tbl as kind, options, minimum and maximum, so the second screen restates what a set would say.
  • The Setup block holds 54 more parameters from the ORE input files, such as outputPath, csvSeparator and baselTrafficLightConfig. No screen draws them and the contract states their domains, so a set would be a rule nothing reads. Three of them do name a catalogue vocabulary (dateCalendar, dateConvention, baseCurrency) and are a candidate for a later task.

4.1.5. What this task changes

  1. Add the setup sets: generate_value_sets.py reads the engine settings groups' curation and writes one set per setting (flag is a boolean set, a whole with bounds is a numeric set with minimum and maximum, a choice is an enumerated set), linked at scope Setup by the ORE name. The numeric kinds gain bound enforcement in the same function the parameter trigger uses, because a whole setting states a floor today and a set must be able to.
  2. Convert the 12 settings the evidence supports to an enumerated or boolean set through a reviewed curation entry. Calibration stays text because the stored population holds Deal, which the guide's list omits, and SeparateCallPutVols stays text because ORE also accepts a regex pattern there; both sets say so in their description.
  3. Read the set on the engine settings screen: the BFF serves the Setup value sets, checks the values a person writes against them, and the screen draws its control from controlOfValueSet() and checks the text from the set. The duplicated columns leave the group settings model.
  4. Retire kind and options from the pair setting model, regenerate every address, and delete the seed, BFF and web code that reads them. The web reads the catalogue of the set instead of the reference kind.
  5. Add build/scripts/check_parameter_value_sets.py and register it in local_checks.py.

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
Pair settings in the seed 834 rows, 126 distinct settings grep -c "^ ('" projects/ores.sql/populate/analytics/analytics_pricing_pair_settings_populate.sql 92fd6990bc85
Pair settings with options 291 rows select count(*) filter (where options <> '') from ores_analytics_pricing_pair_settings_tbl database a4924adbc941bfc7
Kinds in the pair settings number 332, select 291, reference 140, text 67, tenor 4 select kind, count(*) from ores_analytics_pricing_pair_settings_tbl group by kind database a4924adbc941bfc7
Value sets and links 218 sets, 712 links python3 projects/ores.sql/utility/generate_value_sets.py --check 92fd6990bc85
Pricing contract parameters 251 (61 model, 173 engine, 17 global), 0 without a set generate_value_sets.contract_pairs() and resolve() 92fd6990bc85
Setup parameters 15 engine settings, 0 links curation/engine-settings-groups.json and the link rows 92fd6990bc85
Setup block parameters with no screen 54 block_parameter_rows.rows() filtered to Setup 92fd6990bc85
Sets and links after the change 224 sets, 755 links python3 projects/ores.sql/utility/generate_value_sets.py --check a1aa7f6a00
Drawn parameters the gate checks 266 python3 build/scripts/check_parameter_value_sets.py a1aa7f6a00
Settings the guide lists and the code typed as text 15, of which 13 become a set and 2 stay text compare pricing_vocabulary.json options with value-sets-code.json 92fd6990bc85
The build 325 targets, exit 0 ./compass.sh build 7676e1e237
The local gate 54 of 54 checks passed python3 build/scripts/local_checks.py --exclude-heavy f529c48e72
The C++ suites reporting core 970, reporting service 26, analytics core 882, ore core 478353, ore service 49, refdata core 5723, shell reporting 1642, shell analytics 1368, shell refdata 7980 assertions: all pass the suite binaries under build/output/linux-clang-release-ninja/publish/bin f529c48e72
The pgTAP suite 65 files, 623 tests, all pass projects/ores.sql/test/run_tests.sh ea97288e0d
The BFF suite 1144 tests passed npx vitest run in projects/ores.web/packages/bff 7676e1e237
The web suite 2328 tests passed npx vitest run in projects/ores.web/packages/web 7676e1e237

6. Notes

6.1. What changed, file by file

Area Change
curation/engine-settings-groups.json States six Setup sets with provenance, and every group setting names its set instead of a kind, words and a range.
curation/value-sets-guide-enums.json New. States the guide's choices for the 13 settings the code could only read as text, and says in the set why two stay text.
curation/__init__.py Loads the engine settings groups and refuses a setting that states a control beside its set, or names a set the file does not state.
ores.sql/utility/setup_value_sets.py New. The Setup sets and links, read from the curation.
ores.sql/utility/generate_value_sets.py Adds the Setup sets after the curve ones, applies a reviewed file only when it carries its provenance, and gains the guide-enums file.
ores.sql/utility/generate_engine_settings_groups.py Writes only what the group settings table still states.
ores.sql/create/reporting/reporting_ore_parameter_value_validate_fn_create.sql Adds ores_reporting_number_in_bounds_fn so a numeric set's floor and ceiling are checked, and uses it for a range.
ores.reporting/modeling/ores.reporting.engine_settings_group_setting.org Loses kind, options, minimum and maximum.
ores.analytics/modeling/ores.analytics.pricing_pair_setting.org Loses kind and options.
bff/src/engine-settings.ts Serves the Setup value sets beside each setting and checks a written value against its set.
bff/src/ore-value-sets.ts Serves the set's catalogue as well as its kind, values and bounds.
bff/src/pricing-engines.ts Stops serving the pair setting's kind and options.
web/src/parameters/control.ts One control mapping for both screens; the engine settings kind mapping is deleted.
web/src/engineSettings/model.ts Reads the value, its bounds and its words from the set.
web/src/pricingEngines/parts.tsx Reads the sensitivity template heading from the set's catalogue, not a reference kind.
build/scripts/check_parameter_value_sets.py New. The gate.
seeder/generate_vocabulary_sql.py Stops writing the two columns, and gains --from-json so a seed change needs no ORE checkout.
sql/utility/pricing_parameter_rows.py The contract's format prose stops listing the choices; the set states them.
reporting/service/engine_settings_proposal.cpp A proposal asks the store's own value set check, and keeps the whole-number width check the apply makes.

6.2. Where the value set model still cannot state what a per-pair column stated

  • A kind that is also a flag. SeparateCallPutVols accepts true, false or a regular expression inclusion pattern. A set is one kind, so it cannot say "a switch or a pattern"; it stays text and its description says why. The per-pair select column could not state it either, and refused a pattern the guide allows, so nothing is lost here.
  • A vocabulary the evidence states in part. The guide lists Bootstrap, BestFit and None for Calibration, and the documents ORE ships write Deal. The set stays text with the reason. Making it enumerated would refuse a value the seeded population holds, so the task needs the ORE enumeration read from the source before it narrows.
  • A per-pair difference. The per-pair columns could, in principle, state one list for one pair. They never did: every row of a setting carried the same list. The set is keyed by (scope, name) and cannot state a difference by pair. If ORE ever needs one, the link becomes keyed by the pair too; that is a model change and no screen needs it today.
  • A default per pair. usual_value states the value ORE's examples use and is a value, not a rule, so it stays on the pair row. It is not in the set, and a set cannot state it: two pairs may use different example values for one setting.
  • Documentation prose. The format text of a Setup ore parameter still says "A whole number, at least 1." The set is the rule and the prose is documentation a person reads; the pricing contract's prose no longer repeats the choices, and the Setup prose could follow in the documentation task.

6.3. Decisions

  • The Setup sets share a code where the rule is shared. Eleven flags take setup.flag, and the two fixing lags take setup.fixing_lag_days, because a set states a rule and the rule is the same. The alternative, one set per setting, would repeat the same rule fifteen times.
  • The numeric bounds moved into the shared check. A whole setting stated a floor in the group settings row. The value set states it now, and the function the parameter trigger calls applies it, so a bound is data and one function reads it.
  • The proposal service asks the database rather than re-reading the tables. The engine run settings proposal path had its own copy of the rule. It asks ores_reporting_value_in_set_fn now, so a proposal is refused with the sentence the store would use, and the whole-number width check the apply makes stays with it.
  • The check reads the curation and the seed, not a live database. The gate must run on a pull request, where no environment exists. A stale seed is caught by comparing the curation with the committed script.
  • The Setup block's 54 input-file parameters get no set. No screen draws them and the contract states their domains. dateCalendar, dateConvention and baseCurrency do name a catalogue vocabulary and are a candidate for a later task; they are recorded here rather than decided silently.

6.4. The live proof, as a person repeats it

  1. compass db recreate -y -k, then the ACME provisioning script.
  2. npx tsx tmp/prove-value-sets.ts in projects/ores.web, signed in as tenant_admin@acme_corporation (or adrian.vance@acme_corporation for the two vocabularies alone). It prints each setting's set, kind, choices and bounds, and exports and previews one pricing engines document.
  3. Add a value to setup.observation_model and raise the floor of setup.n_threads in the database, then run it again: the choices and the floor follow the set.
  4. npx tsx tmp/prove-write.ts shows the group write refused below the floor.
  5. update ores_analytics_pricing_model_product_parameters_tbl ... with a value outside calibrationstrategy is refused by the trigger, naming the value, the setting and the set.

The two scripts are scratch, beside the task rather than in the tree: they sign in to a live environment and are not part of a test suite.

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

8. PRs

PR Title
#4102 [agile] Close the parameter model task
#4090 [reporting] Make the value set the one statement of a parameter's rule

9. Review

Comment summary File Decision Notes
       

10. Result

The parameter data model is the one source of a parameter's control and rule, on both configuration screens, and a check holds it.

  • The value set tables state every parameter's allowed values. 266 parameters a screen draws carry 1 link each, and the 224 sets state an enumerated list, a catalogue, a range, a regular expression, a switch or text.
  • The Setup scope joined the contract. Six sets state the fifteen engine run settings' controls, the group settings table lost its copy, and the screen draws and checks from the set through the same fields the pricing screen uses.
  • The per-pair kind and options columns are gone from the model, the SQL, the C++, the protocol, the TypeScript, the shell recipe and the seed. The compiler found the last reader, a hand-written proposal check, and it asks the store's own function now.
  • Fifteen settings the guide states a vocabulary for stopped being free text: thirteen carry the guide's list, and two stay text with the reason in the set.
  • build/scripts/check_parameter_value_sets.py runs in the local gate. It fails on a parameter a screen draws with no set, on a set whose own kind cannot hold its rule (a floor above its ceiling, an empty or duplicated list, an unregistered catalogue, a patternless expression), on a bound or a list written as a column check on an engine settings part, and on a read of the retired per-pair column. Each was proved by breaking it and reading the failure. The seeded script is compared with its sources exactly by generate_value_sets.py --check, which owns that test.
  • The engine run settings' rule lives only in the set: the part tables lost their column checks and carry a guard that asks the verdict function, and the ORE run document codec lost its own list of observation models.
  • ore_parameter_settings check reports the 266 parameters the screens draw, the engine run settings among them, and answers OK when every one names a set of a kind its domain takes. It refuses to judge a database whose ORE parameter contract is not published, and says so.

Proven live on a recreated, provisioned environment: adding a value to setup.observation_model in the database added it to the screen's choices, and raising the floor of setup.n_threads moved the bound the server enforced, with no server change. A group write below the floor was refused. A pricing engines document exported and previewed with 0 rows added, changed or removed and 0 issues. A parameter write outside its set was refused by the trigger, naming the value, the setting and the set. The retired columns appear in no reply.

Not done here, and stated rather than hidden: the Setup block's 54 input-file parameters have no set because no screen draws them; three of them name a catalogue vocabulary and want a later task. Calibration stays text until the ORE enumeration can be read from the source, because the guide's list omits a value the seeded documents write. The pending provenance columns on the row are the story's own task, not this one.

Commits on the branch, in order: the inventory, the Setup sets, the guide's choices, the engine settings screen reading the set, the retired per-pair columns, the gate, and the proposal check.

Emacs 29.3 (Org mode 9.6.15)