Task: Make the scheduler-triggered report execution correct
Table of Contents
- 1. Goal
- 2. Status
- 3. Acceptance
- 4. Analysis
- 4.1. The headline: a schedule never creates a report instance
- 4.2. The FSM: seven states seeded, five written, one enforced by nothing
- 4.3. The engine: the pipeline is far more complete than its plan claims
- 4.4. Defects in the engine as the report workflow uses it
- 4.5. Other findings on the path
- 4.6. How to reproduce the live experiment
- 5. Progress
- 6. Live verification
- 7. The chain the live run exposed
- 8. The chain now walks
- 9. Plan
- 10. Decisions
- 11. Notes
- 12. Test Scenarios
- 13. PRs
- 14. Review
- 15. Result
This page documents a task in the Clean ores.reporting to the component clean standard story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
A schedule fires, the reporting service creates a report instance in the correct state, the report execution workflow runs to completion, and the instance reaches a terminal state with an accurate output_message. Every FSM state and transition the model seeds is either implemented or recorded as unimplemented with its reason.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Clean ores.reporting to the component clean standard |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-09-26 |
3. Acceptance
- A scheduled report definition fires, and a
report_instancerow appears in the state the concurrency policy demands. - The report execution workflow runs to completion, and the instance reaches
completedwith an accurateoutput_message. - Every failure path answers the caller, or records why it cannot.
- Every one of the seven seeded FSM states is either written by code or recorded as unimplemented with its reason.
- A test fails when the trigger path breaks.
4. Analysis
The section below is the answer to "how much of the report FSM is implemented". Every claim carries its evidence. The live experiment is reproducible from the steps in the last subsection.
4.1. The headline: a schedule never creates a report instance
The scheduler-to-report path is dead. Proven live on 2026-09-26, not merely read:
- A
report_definitionand a scheduler job were inserted, the job publishingreporting.v1.report-instances.triggerevery minute. - The scheduler fired and reported success —
Firing job: reporting.v1.fsm-live-probethenNATS publish action succeededthenJob succeeded. - The reporting service logged
Handling ...reporting.v1.report-instances.triggerand then nothing. select count(*) from ores_reporting_report_instances_tblreturned 0.
The cause is one missing header. ores.scheduler publishes the trigger
with client::publish and no headers
(nats_publish_action_handler.cpp:79). make_request_context takes the
!verifier early return only when no verifier exists, and the reporting
service always builds one from the IAM JWKS key
(domain_service_runner_impl.hpp:94, request_context.cpp:133). So the
trigger is rejected unauthorized.
The rejection is then thrown away. error_reply returns immediately when
the message has no reply subject (handler_helpers.hpp:220-224), and a
fire-and-forget publish has none. The scheduler therefore records the
job as succeeded. Nothing anywhere reports the failure.
Three further defects sit on the same path, all currently unreachable because the auth check runs first:
| # | Defect | Evidence |
|---|---|---|
| 1 | The concurrency check asks "does any instance exist in this tenant", not "does an instance of this definition run". list_instances(0,1) filters tenant, workspace and current version only. |
report_instance_trigger_handler.hpp:99, report_instance_repository.cpp:164 |
| 2 | A queued instance never runs: nothing performs queued -> pending. |
no writer of pending outside the trigger |
| 3 | started_at is stamped on skipped, queued and failed instances, which never started. |
report_instance_trigger_handler.hpp:127 |
4.2. The FSM: seven states seeded, five written, one enforced by nothing
The machine report_instance_lifecycle seeds 7 states and 7 transitions
(dq_fsm_report_instance_lifecycle_populate.sql).
| State | Written by | Reachable in practice |
|---|---|---|
| pending | trigger | no — the trigger is rejected |
| queued | trigger (policy queue) |
no, and no promotion to pending exists |
| running | gather_trades step |
no — needs a dispatched workflow |
| completed | finalise step |
no |
| failed | trigger, mark_instance_failed, fail handler |
no |
| cancelled | nothing | dead state |
| skipped | trigger (policy skip) |
no |
Two conclusions follow.
- The transitions table is enforced by nothing.
ores_dq_fsm_transitions_tblis seeded for this machine and read by no code. A caller assignsinst->fsm_state_iddirectly (report_execution_handler.cpp:96,175,380), and the generated CRUD handler accepts any state a client sends. The only consumer of the FSM tables isget_fsm_states_request, which returns names and UUIDs. So an illegal transition is unrepresentable in the tables and unconstrained in the code. - The granular states the design proposes do not exist.
2026-04-05-report-execution.orglists 11 states (gathering_inputs,preparing_package,submitted_to_compute,computing,processing_results). None is in the SQL. The implementation plan deferred them deliberately, on the argument thatworkflow_stepalready gives per-step visibility, and narrowed the instance FSM topending -> running -> completed/failed. That is a decision, not an omission, and this task keeps it unless the decision is revisited.
4.3. The engine: the pipeline is far more complete than its plan claims
2026-04-06-report-execution-workflow-implementation.org marks phases
3.8 to 3.11 DEFERRED. The code says otherwise.
| Step | Owner | State |
|---|---|---|
| gather_trades | ores.reporting | implemented |
| gather_market_data | ores.reporting | implemented |
| assemble_bundle | ores.reporting | implemented |
| prepare_ore_package | ores.ore.service | implemented (download, tar, upload) |
| submit_compute | ores.compute | implemented, completes asynchronously |
| collect_compute_results | ores.reporting | stub — pass-through |
| finalise | ores.reporting | implemented |
The definition is registered, not orphaned:
ores.workflow/core/src/messaging/registrar.cpp:106 calls
register_report_execution_workflow, and that registrar is reached from
the workflow service's own application.cpp. Every command subject has
a subscriber.
Step 4 is the interesting one. submit_compute deliberately does not
complete its own step. It creates a batch and its workunits, dispatches
the work assignments, writes a workflow_batch_link, and returns
(report_submit_handler.cpp:133-143). ores.compute's
batch_workflow_bridge publishes step_completed_event when the batch
closes. The async wait therefore lives outside the engine, which has no
wait_for_event primitive and no step timeout. If that bridge does not
run, the report hangs in in_progress forever and nothing notices.
4.4. Defects in the engine as the report workflow uses it
| # | Defect | Evidence |
|---|---|---|
| 1 | No step timeout. A missing or dead bridge hangs a report indefinitely. | engine has no timeout; see above |
| 2 | The failing step is never compensated, and step 0 failing dispatches no fail_report at all. The instance is still marked failed only because gather_trades calls mark_instance_failed itself. |
workflow_engine.cpp:276; report_execution_handler.cpp:145,160,191 |
| 3 | completed_with_warnings steps are never compensated. |
workflow_engine.cpp:276 |
| 4 | Every step shares one compensation subject, so N completed steps publish N identical fail_report commands. Idempotent, but noisy, and the engine does not dedup. |
report_execution_workflow.hpp:71,108,145,198,253,298,343 |
| 5 | The workflow notifies nobody of completion; the report is finalised by its own last step instead. | engine publishes workflow_instance_changed only |
| 6 | No test names any workflow type. The definition, its command builders and its compensation are untested. | tree-wide grep |
4.5. Other findings on the path
gather_market_dataexports every series for the tenant; nothing filters by trade type or date. The plan records this as a future enhancement.- The definition lifecycle FSM has the same shape of gap:
ores_reporting_initial_definition_state_fn()has no caller, sofsm_state_idis left NULL on a new definition, andreconcile()schedules every definition with noscheduler_job_id— including ones that should be archived. - The trigger uses
publish, notjs_publish, so a trigger fired while the reporting service is down is lost with no record. - One scheduler firing produced two
Handlinglines 103 ms apart in the reporting log. The cause is not yet established. It matters, because a doubled trigger would double-create instances once the auth check passes. Verify before the fix lands.
4.6. How to reproduce the live experiment
- Insert a
report_definitionin the system tenant withconcurrency_policy'fail'= andschedule_expression'* * * * *'=, usingadminas =modified_by~ (the insert function validates it). - Insert a scheduler job with
action_type'nats_publish'= andaction_payload={"subject":"reporting.v1.report-instances.trigger", "report_definition_id":"<id>","tenant_id":"<tenant>"}. compass services restart ores.scheduler.serviceso the loop reloads jobs.- Raise
ORES_SERVICE_LOG_LEVELtodebugand restartores.reporting.serviceto see theHandlingline. - Wait for a minute boundary. Read the scheduler log, the reporting log,
and
ores_reporting_report_instances_tbl.
5. Progress
| Unit | State | Evidence |
|---|---|---|
| Test report fixture | Landed. Five definitions covering the happy path, the three concurrency policies and a config with no scope. Idempotent; a second run creates nothing. | reporting_test_report_populate.sql |
| Component regenerated | Landed. 54 files brought up to date and 20 materialised. | check_component_drift.py --component reporting |
| Trigger as an operation | Landed. The generated protocol retires the hand-written message, and the shell gains report_operations trigger-report-instance <report_definition_id> <tenant_id> [--job_instance_id <v>]. |
report_operations_protocol.hpp |
| FSM transitions enforced | Landed and proven against the live database. Six checks, including two refusals. | reporting_report_instances_fsm_guard_create.sql |
| In-flight query | Landed. An instance is in flight while its state is not terminal, read from the machine rather than from a list of names. | reporting_in_flight_instance_fn_create.sql |
| Scheduler identity | Landed. The scheduler builds a service token and sends a report trigger as an authenticated request, so a refusal fails the job. | nats_publish_action_handler.cpp |
| Handler rewrite | Landed. The tenant comes from the request, the concurrency check reads the in-flight query, an unknown policy is refused, started_at is set only when the instance will run, and every path answers. |
report_instance_trigger_handler.hpp |
Still open on this task:
queuedtopendingpromotion. The transition exists and the guard enforces it, but nothing performs it, so a queued instance waits forever. It belongs where an instance reaches a terminal state.cancelledis written by no code.- The trigger's concurrency check and the instance insert are not one transaction, so two simultaneous triggers can both see nothing in flight.
- The generated CRUD path can still write
fsm_state_id, though the guard now refuses what the machine forbids.
6. Live verification
Run through the shell against the fleet on 2026-09-26, with the fixture
definition FSM Happy Path:
report_operations trigger-report-instance 7ddbb72b-... ffffffff-...
-> {"result":{"outcome":"ok","code":"triggered",
"message":"Report instance c7779c02-... created."}}
The instance row exists in state pending, and the reporting log records both
the save and "dispatched its workflow". The workflow service records
Start-workflow request received: type=report_execution_workflow.
The run also exposed three defects that only a live run could find, each fixed in the same commit: the handler scoped its context from the base context and so lost the workspace, which made every definition look missing; the in-flight query was revoked from PUBLIC and granted to nobody, so the trigger failed with "permission denied"; and the change-reason code was an unseeded literal, so the insert was rejected by the validator.
What remains is the step the run stopped at: the instance does not leave
pending. The engine accepts the start and the reporting service never logs
gather_trades, so the next diagnostic is whether the engine published the
step-0 command and to which subject.
7. The chain the live run exposed
Driving the trigger from the shell one defect at a time reached a definite answer to the second half of the objective, and the answer is that the report service does its part correctly and the chain broke behind it. In order:
| # | Where | Defect | State |
|---|---|---|---|
| 1 | ores.scheduler | Published the trigger with no identity, so the reporting service rejected it unauthorized and the job was recorded as succeeded. | Fixed |
| 2 | ores.reporting | The handler scoped its context from the base context and lost the workspace, so every definition looked missing. | Fixed |
| 3 | ores.reporting | The in-flight query was revoked from PUBLIC and granted to nobody. | Fixed |
| 4 | ores.reporting | The change reason code was an unseeded literal. | Fixed |
| 5 | ores.workflow | No workflow instance has ever been created in this environment: both workflow tables require modified_by, validate it, and the engine set only created_by. Every start failed, so no step was ever dispatched. |
Fixed |
| 6 | ores.reporting | The instance took the definition's name and names are unique per tenant, so a second run of one definition was refused. | Fixed |
| 7 | ores.reporting | The definition lookup alternates between finding the fixture definition and reporting it missing, from one request to the next, in one process. The lookup filters on workspace_id = ctx.workspace_id() and the party policy scopes by app.visible_party_ids; the fixture's party is ACME UK while the admin session's party is the System Party. Next diagnostic: read context::with_tenant and the pool's session-variable reset to see whether a pooled connection can carry a stale app.visible_party_ids from another request. |
Open |
The concurrency policy was observed working: with a stale pending instance in
flight, the next trigger against the fail policy created its instance already
failed rather than dispatching it, which is what the seed says it should do.
8. The chain now walks
After defects 1 to 6 were fixed, one more surfaced and then the FSM moved.
| # | Where | Defect | State |
|---|---|---|---|
| 7 | ores.reporting | The fixture was owned by an arbitrary business party, so a session scoped to another party could not see the definition it was meant to trigger. The fixture is now owned by the tenant's system party, which every session can see. | Fixed |
| 8 | ores.reporting | ores_reporting_resolve_book_ids_for_config_fn compared valid_to against the literal 'infinity', while every generated table stores ores_utility_infinity_timestamp_fn(), which is 9999-12-31 23:59:59+00. It therefore resolved no scope for any config, and every report failed with "no books or portfolios in scope". |
Fixed |
| 9 | environment | The trading service performs export_trades_to_storage, and it has no HTTP base URL: only ORES_ORE_SERVICE_HTTP_BASE_URL is defined, the trading unit passes no --http-base-url, and its parser declares the option. The upload therefore cannot connect, so no report can gather trades. |
Already captured, outside this component: Three services default their storage base URL to a port the server does not use. ores.trading, ores.marketdata and ores.reporting all default the flag to port 8080 and their units pass no override. |
Observed end to end after defect 8, through the shell:
trigger -> outcome ok, code triggered, instance 6423515a-... created instance -> pending, then running, then failed message -> export_trades_to_storage failed: connect: Connection refused
So pending -> running -> failed now runs, the guard accepts each transition,
and the failure is a real one carried into output_message. The remaining
blocker is defect 9.
9. Plan
- Seed a test report in the component, so the FSM is testable in any
environment. A definition, its
risk_report_configand its book scope. - Make the trigger an operation and give it an identity, per the decisions below.
- Enforce the seeded transitions, so an illegal state change fails.
- Walk the FSM from ores.shell and keep that walk as the test.
- Correct the two plan documents, whose status tables no longer describe the code.
10. Decisions
10.1. The trigger is a service-to-service operation, not an entity verb
Authentication here has two layers, and the defect lives on the second.
- Transport is mTLS.
build/config/nats-<label>.confsetsverify: truewith the environment CA, so every client presents a certificate, one per service. There are no subject ACLs. Being able to publish to a subject means being a process in this environment. - Message identity is the JWT bearer. It carries the tenant, the party
scope and the roles, and those are what row-level security and
has_permissionread.
So the scheduler is not refused entry to the bus. Its message arrives and names no principal, and the handler demands one.
Three kinds of subject already exist:
| Kind | Shape | Context |
|---|---|---|
| Entity verbs | {component}.v1.{resource}.{verb} |
Built from the bearer |
| Operations | {component}.v1.ops.{operation} |
Built from the bearer; the tenant comes from the token, and an acted-on party is carried in the message |
| Control plane | workflow.v1.start, workflow.v1.events.step-completed |
No context; the engine trusts the bus |
trigger is not a canonical entity verb, so P01 and P04 place it in the
second row. It is also a scheduler-to-reporting call, and every other
service-to-service call in the tree authenticates with the caller's own
service token: make_service_token_provider logs in at
iam.v1.auth.service-login, caches the bearer, refreshes it, and
authenticated_request attaches it. Reporting already calls trading,
marketdata and the scheduler that way.
The fix therefore has four parts, and they are not alternatives:
- Modelling. The trigger becomes an operation model under
reporting.v1.ops., and its message generates from the model. The hand-written protocol header goes, as P04 requires. Variability'sores.variability.operations.orgis the worked example. - Identity. The scheduler publishes with its own service account.
ores_<env>_scheduler_servicealready exists, so this follows the existing service-call pattern and gives the audit trail a name. - Tenancy. The message carries the acted-on
tenant_id, because one service account lives in one tenant while report definitions are per-tenant. This is the precedentcomplete_party_onboardingset for the party: the acting identity is the caller's, the acted-on identity is carried. - Failure. The scheduler sends the trigger as a request, so a reply
subject exists and
error_replystops being discarded. A rejected trigger then marks the job failed instead of succeeded.
10.2. The seeded transitions are enforced
An instance state change goes through a checked transition against
ores_dq_fsm_transitions_tbl, so an illegal transition fails loudly. Today
inst->fsm_state_id is assigned directly and no code reads the table.
10.3. A seeded test report makes the FSM testable
The component carries a test report — a definition, its
risk_report_config and its book scope — so that an environment, freshly
created, can walk the FSM from ores.shell. The fixture is deterministic: a
book with no trades exports an empty list and succeeds
(trade_handler.hpp:364-371), so the happy path needs no trade data.
The fixture covers the happy path pending -> running -> completed; a
configuration with no book scope, which gather_trades rejects and which
drives -> failed; and the three concurrency policies, exercised by
triggering one definition twice.
11. Notes
- Baseline (B01) for the parent task:
check_component_drift.py --component reporting --dry-runreports 54 files that would change and 20 that would be created — 74 in total. Saved at.audit/reporting/B01-drift-dryrun.txt. - ores.reporting is not in
COMPONENTS_UNDER_TEST, so no codegen gate covers it today.
12. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
13. PRs
| PR | Title |
|---|---|
| #2212 | [reporting] Bring ores.reporting to the Component Clean Standard |
14. Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|
15. Result
Delivered. The report instance FSM is implemented, enforced, and proven.
Ten defects stood between a scheduled trigger and a completed report. Nine were outside ores.reporting: the scheduler published with no identity and the rejection was discarded; the workflow engine set no modified_by, so no workflow instance had ever been created; the storage server's bucket allow-list omitted report-data; ores.ore moved a base URL out from under the package handler with a std::move; ores.compute assigned a nil application version; and the environment carried stale notify triggers and no market-data base URL. Reporting's own code needed three fixes, and none of them was why the pipeline could not run.
Proof: a report triggered from the shell runs the seven-step workflow to completed, with the instance travelling pending, running, completed through the database guard. Witnessed twice – once on the database as it had drifted, once on one built from nothing.
The review of PR #2212 raised three findings, all fixed: the scheduler paths saved a null FSM state silently after the job had been created or deleted; compensation overwrote the true failure reason with an unrelated step's; and the concurrency policy codes were inline literals.
Not delivered. The eight execution-step shell commands cannot answer outside a workflow, because the handlers return in silence when the workflow headers are absent – diagnosed, and the fix is a shared handler contract rather than this component's.
One caveat on the environment. The completed walk depended on two ORES_*_SERVICE_HTTP_BASE_URL variables added to .env by hand, which is why the durable fix belongs in the environment profile and not in that file.