Task: Produce the engine's input from the entities

Table of Contents

This page documents a task in the Model the ORE report configuration as entities story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

A run's engine input is generated from typed entities, validated against the ORE schemas, and packaged with the files the run document names.

2. Status

Field Value
State DONE
Parent story Model the ORE report configuration as entities
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-09-27

3. Acceptance

  • The generated archive contains Input/ore.xml and every configuration file it names. That the engine runs the archive moved on 2026-10-04 to Write the portfolio and market data files into the engine input, since it needs the portfolio and market files too.
  • No field is written as a name/value pair or a blob: every value comes from a typed column.
  • The generator is tested against the vendored example bundle.

4. Plan

Import a run's configuration from an ORE input directory built the export: a report definition's run document and the documents it binds, rebuilt as files. This task puts that export into the package step, so the archive the compute wrapper unpacks carries the run's input.

  1. prepare_ore_package_request carries the report definition id, which the workflow already holds in its start request.
  2. The ORE service's package handler exports the definition under the run's tenant and writes the files into the archive with archive_layout.
  3. export_run narrows itself to the definition's party when the session has none, since a workflow step runs with the tenant alone and would otherwise read every party's conventions.

Decisions:

  • The engine is started on Input/ore.xml, so the run document goes there; every other file goes under the run's inputPath, as ORE reads them. The corpus sets inputPath to Input or a folder under it in every run document.
  • The trades and market data stay the msgpack blobs the earlier steps write. Turning them into the portfolio and market files the run document names is separate work, so the engine cannot run the archive yet.

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
       

6. Notes

Today's market: how a run reaches its document, and an open question. Recorded in the PR #2500 review. A run names its today's market file in report_run_setup.market_config_file (from marketConfigFile in ore.xml), and binds each analytic stage to a configuration id in report_market_binding (from the Markets block). todays_market_config.configuration_id links the document to its reporting.configurations header, which is what joins the run to the rows.

Inside the document, every reference is text. A todays_market_configuration_binding.reference of inccy for DiscountingCurves means the DiscountingCurves collection whose collection_id is inccy. Nothing checks it. This task is the first code that resolves the chain, so it decides whether the binding should carry a todays_market_collection_id foreign key, or whether name resolution at export time is enough.

The export resolves the chain by name, as ORE does, and nothing checks it at write time. The decision moved to its own task on 2026-10-04: Link today's market bindings to their collections.

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

8. PRs

PR Title
#2608 [ore] Package the run's document and configuration for compute

9. Review

# Comment summary File Decision Notes
1 Path traversal: an absolute or .. inputPath or file name writes outside the stage directory run_store.cpp Accept archive_layout refuses an absolute path, a root name or a .. component, for the input path and each file; tests cover ../, an absolute path and a file name.
2 Actor and service go into set_config unescaped, now on every acquire tenant_aware_pool.hpp Accept The pool doubles quotes; the raw query helpers, which had the same defect, bind the value. A pooled write and a raw read each carry an actor holding an apostrophe.
3 inputPath is not normalised run_store.cpp Accept Layout keys are lexically normal; a test covers ./Input//Dim/.
4 CI drift: the generated shell script lacked the new field report_operations-prepare-ore-package.ores Accept Regenerated with tangle_shell_scripts.
5 Self-found: the first pool test ran through the raw query helpers, which open a fresh connection each call, so it never exercised the pool session_settings_tests.cpp Fix Replaced by pooled_session_tests.cpp, which alternates party B and a tenant-only session over fifty repository reads and requires the tenant-only session to see party A's row.

10. Result

The compute package now carries the run's input. The package step exports the report definition through the run store and writes Input/ore.xml and every configuration file it names into the archive, under the run's inputPath.

  • prepare_ore_package_request carries the definition id, from the operations model, so the shell command and the web protocol carry it too.
  • export_run narrows a session with no party to the definition's party, and archive_layout places the files.
  • Building the narrowing test exposed a defect in the connection pool: a context without a party, visible parties, actor or service skipped those settings, so a reused connection kept the previous context's. A tenant-level context could see another party's rows and stamp another actor. The pool now writes every setting on every acquire, a test proves it over fifty pooled repository reads, and four refdata tests that passed only through the leak now build their repository after setting the party.

The engine cannot run the archive yet: the portfolio and market data are still msgpack files, and Write the portfolio and market data files into the engine input, owned by another environment, closes that.

Emacs 29.3 (Org mode 9.6.15)