Task: Bring ores.reporting to the clean standard
Table of Contents
This page documents a task in the Clean ores.reporting to the component clean standard story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
ores.reporting passes the Component Clean Standard for a component of kind Entity.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Clean ores.reporting to the component clean standard |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-09-23 |
3. Acceptance
- Every checklist item that applies to a component of kind Entity passes, with its evidence recorded on the task.
- Every item that does not apply is recorded as not applicable, with the reason.
- Every generator defect found is fixed at its source, with a regression test.
- ores.reporting is listed in
COMPONENTS_UNDER_TEST, andcheck_component_drift.py --allpasses.
4. Before the rebase
The branch is 140 commits behind main, and a merge-base check over the files this pass fixed shows that two of its fixes have been made upstream independently. They are not conflicts to resolve in this branch's favour:
| File | Upstream | What it means |
|---|---|---|
ores.workflow/.../workflow_engine.cpp |
73442db01d [workflow] Set modified_by so that a workflow can start |
The same defect this pass fixed in round 4. Take upstream's version. |
ores.http/.../storage_routes.cpp |
eb3331ca63 [storage] Authenticate the storage routes, add HEAD and listing, and share one store |
It deletes the k_known_buckets allow-list rather than adding report-data to it, which is the better fix. Take upstream's version. |
ores.codegen/scripts/component_registry.py |
one upstream commit | Likely a registry entry that conflicts with adding reporting; re-apply the entry on top. |
That two of this pass's ten defects were found and fixed upstream while it ran is its own evidence: they were real, they were not local, and the merge is the point at which the duplicate work is discarded rather than carried.
5. Handoff
State at the end of this pass, verified: the working tree is clean, `check_component_drift.py –component reporting` exits 0, and all four reporting suites pass.
The objective's second question is answered, twice over. A report triggered from the shell runs the seven-step workflow to `completed`, with the instance travelling `pending`, `running`, `completed` and the FSM guard admitting each transition. It was proven once on the database as it had drifted and once on a fresh `db recreate`, so the result does not depend on accumulated state. Ten defects stood in the way, nine of them outside the reporting component.
The checklist has a recorded result for all 49 items. Passing: B01, B03, B04, B06, G01, G02, G05, G07, M01, M02, M03, M09, M10, H05, P03, P04, V01, V02, V03, V05, V06, V07, W01, W02, W03. Partial and measured: V04 (thirteen commands; three families and eight step commands remain, the latter recorded as unable to answer with the cause diagnosed) and V08 (four of six files have their logic under test). Waiting on ores.dq: M06 and P02's remainder.
What to do next, in order.
- Rebase. The branch is well behind main and two conflicts already have a disposition: take upstream's version of `workflow_engine.cpp` (`73442db01d` sets `modified_by`, the same defect fixed here) and of `storage_routes.cpp` (`eb3331ca63` deletes the bucket allow-list outright). Re-apply the `reporting` entry to `component_registry.py` on top of whatever landed there.
- V08's last two files. `report_input_bundle_repository.cpp` needs a database test, following the pattern of the existing eventing integration tests.
- V04's tail. The three remaining families are template repetition of the one already exercised; the eight step commands cannot answer outside a workflow and need the handler contract changed to reply `bad_request` instead of returning in silence, which is an `ores.service` change and not this component's.
- Captures worth filing, each with its evidence on this task: the twin gate cannot see a wire type declared outside a `*_protocol.hpp`; nothing checks that a declared subject has a subscriber (twenty did not); the populate scripts cannot be replayed against a populated database because they pass `current_user`; the generated per-entity registrar family exists in twelve components with no caller; and the report instance's `output_message` is overwritten by compensation so it names a step that succeeded.
6. Plan
Work the Component Clean Standard in its section order, one entity at a time, and record each item's result in the table below.
- Baseline and inventory (B01 to B05). Record the baseline before any edit.
- Models (M01 to M10).
- Protocol (P01 to P06).
- Generation (G01 to G06).
- Wiring and data (W01 to W04).
- Shell (S01, S02).
- Structure and hygiene (H01 to H03).
- Verification (V01 to V06), ending with the registry entry.
| Item | Result | Evidence | |
|---|---|---|---|
| B01 | Baseline recorded 2026-09-26: 54 files would change and 20 would be created, 74 in total. | .audit/reporting/B01-drift-dryrun.txt |
|
| V01 | Pass for the component: the whole tree builds, repeatedly, including after every unit in this pass. | cmake --build --preset linux-clang-debug-make |
|
| V02 | Pass. All four reporting suites pass: ctest -R reporting reports 4 of 4, with ores.reporting.core.tests at eighteen cases. The core suite's four failures were the live notify triggers emitting the old payload shape, fixed by applying the regenerated trigger files; the shell suite's was the generator defect G05 named, fixed in the template and the loader. |
ctest -R reporting |
|
| G05 | Pass. The defect is fixed in the template and the loader, the eight IAM shell operation tests regenerated with it, and the codegen-suite regression test is in: the table now covers every token-fillable type – double, an instant, an address and a time point were missing and would have hit the same fallback – and a structural test asserts that no fillable type lacks a literal. The suite went from 649 to 660 cases. | org_loader.py; test_shell_operations.py |
|
| G01 | Pass for the C++ and TypeScript facets. check_component_drift.py --component reporting exits 0 with "No drift: regenerated output matches the checked-in tree" after this pass regenerated 54 files and materialised 20. This row is recorded late: the first attempt to write it silently no-matched its anchor and reported success anyway. |
.audit/reporting/R11-drift.txt |
|
| G07 | Pass for this component: no generator defect was worked around, and every model change made after a regeneration was regenerated rather than hand-edited. | ||
| B03 | Pass. 34 hand-written C++ files remain under src/ and include/, none of them in tests/. The check that matters is that no hand-written file has a generated twin: for every one of the 34, no file of the same name elsewhere in the component carries the generated marker or its *.mustache template output. So none of them is a generatable file left hand-written, and each is infrastructure by elimination: the component registrar, the operations, trigger, scheduling and execution handlers, the publish-from-dq handler, the report input bundle repository, the scheduling service, the service's config and application files, and the shell part's hand-authored CMake. |
find plus a marker check over every hand-written file |
|
| B06 | Pass for the public headers, one finding, now deleted. The census counts, for every hand-written header, the production files that include it; the include-root path is the key, not the file path, which is what a first attempt got wrong. Of the 34 hand-written files only three had no production includer, and two of those are the namespace documentation headers ores.reporting.core.hpp and ores.reporting.service.hpp, which G08 exempts because an empty namespace is their whole content. The third, service/include/ores.reporting.service/app/application_exception.hpp, was dead: nothing included it, while the same file in ores.dq and ores.synthetic is included by each component's own application. It is deleted, the service's CMake list regenerated, and the build is clean. A per-symbol census of the methods inside each surviving class is finer than this pass went. |
grep -rl <include path> projects/ --include=*.cpp --include=*.hpp |
|
| M09 | Pass. The widget column is gone from the detail-fields table of all four models that carried it – concurrency policy, report type, report definition and report instance. Removing it changed no generated output at all: the regeneration rewrote the four models and nothing else, which is the proof that the column was dead rather than merely unused. The model gate still reports no drift. |
git status after check_component_drift.py --component reporting shows the four models and no generated file |
|
| M10 | Open. The root component_overview.org carries only Summary, Sub-components and Entity modules. It lacks the six sections H05 requires, and it has no Diagram section. The three part overviews are complete. |
projects/ores.reporting/modeling/component_overview.org |
|
| V03 | Pass. compass db recreate -y -k completes in 43 seconds, and the fleet comes back with 24 running and none stopped or missing. The recreate is what proves this pass's SQL durably rather than by hand: the notify trigger emits the new action payload shape, ores_reporting_report_instances_fsm_guard_trg exists, ores_reporting_in_flight_instance_fn exists, the fixture seeds its five definitions, and the machine seeds eight transitions with failed an initial state. All four reporting suites then pass against the fresh database. |
.audit/reporting/ run of compass db recreate -y -k |
|
| M06 | Waiting on ores.dq, and that is the finding. The surviving wire type carries std::vector<ores::dq::messaging::dq_report_definition_template>, and that type cannot yet be modelled here: projects/ores.dq/api/include/ores.dq.api/messaging/report_definition_template_protocol.hpp carries no generated marker, so it is hand-written in DQ, and no TypeScript file under generated/dq mentions a template, so it has no projection. Mapping a field to a TypeScript type that does not exist is impossible, and opting the whole operations model out of TypeScript would cost the trigger its twin, which the gate now checks. So reporting's last hand-written wire type waits on ores.dq modelling and projecting the type it re-exports. The correction row follows. |
||
| M06 | (superseded) Corrected, not a pass. One hand-written wire type survives, and the gate that would have caught it cannot see it. report_definition_template_handler.hpp:37-48 declares get_report_definition_templates_request and its response, with the subject reporting.v1.report-definition-templates.list. It lives in a handler header rather than a *_protocol.hpp, so check_protocol_twin_coverage.py never inspects it: the gate pairs generated protocol headers with their twins and has no sight of a wire type declared anywhere else. This is the gate's blind spot as much as the component's defect. The response carries std::vector<ores::dq::messaging::dq_report_definition_template>, so the message is a DQ payload re-exported by reporting, which is the same ownership question P03 raises about the two foreign subjects. Modelled nowhere; the previous row overstated the position. |
||
| M06 | Both hand-written protocol headers were retired in this pass: report_scheduling_protocol.hpp and report_execution_protocol.hpp. With the component listed under its real slug, check_protocol_twin_coverage.py now reports all 45 generated protocol headers in 8 components with a twin, and reporting is one of them. The superseded row follows. |
check_protocol_twin_coverage.py |
|
| M06 | (superseded) Open, and the gate names it exactly. With reporting listed, check_protocol_twin_coverage.py fails on two headers and no others: report_execution_protocol.hpp and report_scheduling_protocol.hpp. Both are hand-written protocol headers, so neither has a model and neither has a TypeScript twin. This is the component's largest remaining gap and it is the same defect as P04. |
check_protocol_twin_coverage.py |
|
| P02 | Open, one of two violations closed. (1, fixed) the registrar's publish-from-dq literal is now a subject-only trigger message in the operations model and the registrar references its constant; the whole tree builds and drift is clean. The remaining violation follows. |
report_operations_protocol.hpp |
|
| P02 | (superseded) Open, exactly two violations, both verified this round. (1) core/src/messaging/registrar.cpp:250 subscribes the literal reporting.v1.report-definitions.publish-from-dq rather than a generated constant; the earlier inventory filtered this one out by pattern and so under-reported P02. (2) report_definition_template_handler.hpp:40 declares its subject in a hand-written struct, which is the same defect M06 records. Both close in one unit: each needs its message in the operations model, and (1) needs a subject-only trigger message, as ores.iam.account_messages already does for iam.v1.accounts.publish-from-dq. |
grep -rn '"reporting\.v1\.' projects/ores.reporting/core |
|
| P03 | Both mechanical halves pass; the ownership question is recorded as a cross-component move. There is no duplicate subject: every one of the 44 the component declares appears once, in the model that generates it. And there is no declared message that nothing serves: the entity verbs, the two scheduling operations, the trigger, the eight execution steps and the templates read are each subscribed by the component that declares them. What remains is the opposite of a duplicate: compute.v1.report.submit and ore.v1.report.prepare-package are declared only in reporting's operations model, and a grep shows compute and ore declare neither. The components that serve those subjects should own them, and reporting should reference their declarations – a move across three components, recorded here rather than attempted from a reporting branch. It is a layering defect, not a wire defect: the generated constants are consistent today because there is only one declaration per subject. |
=grep -rn 'compute.v1.report.submit\ | ore.v1.report.prepare-package' projects/ –include=*.hpp –include=*.org= |
| P03 | (superseded) Open, three questions the B04 inventory raised: the two spellings, the two foreign subjects, and whether every declared subject has a server. The report*-templates subject is served by the component; the two foreign ones are sent, not served. |
||
| P04 | Pass. Every message the component exchanges generates from a model: the trigger, the two scheduling operations, the eight execution step pairs and the workflow payload. No hand-written protocol header remains. | projects/ores.reporting/api/include/ores.reporting.api/messaging/ |
|
| V05 | Open, and the trial run is the evidence. Adding reporting-cpp to COMPONENTS_UNDER_TEST and re-running the two fastest gates changed neither: the twin check still covered 7 components and the permission check still 4. So listing the component is not yet the lever it looks like. The twin gate includes a component only when _component_dir_names(component) intersects the registry and _committed_protocol_dirs reports a committed TypeScript facet for it, so reporting's committed TS protocol is not being picked up. That is the next thing to understand, before any registry entry is claimed. |
check_protocol_twin_coverage.py:126-139 |
|
| V06 | Not claimed. The first trial used reporting-cpp, which never intersects the catalogue's own name, so nothing changed; the second used reporting, which is the right slug and immediately failed the twin gate. The standard says an item that is merely unfinished is not an exception, so the entry is reverted and the component waits. |
component_registry.py |
|
| B04 | Pass. 44 subjects are declared across the generated protocol headers, and the survey found three things. (1) One raw literal survives: report_definition_template_handler.hpp:40 states reporting.v1.report-definition-templates.list as a string rather than referencing a generated constant, which is P02's exact defect. (2) Two resource spellings coexist: every generated entity subject uses snake_case (reporting.v1.report_definitions.list) while the hand-written scheduling, template and workflow-step subjects use kebab-case (reporting.v1.report-definitions.schedule, reporting.v1.report.gather-trades). The spec fixes one spelling for {resource}, so one group is wrong. (3) Two foreign subjects are declared here: compute.v1.report.submit and ore.v1.report.prepare-package are compute's and ore's subjects, referenced by this component's workflow definition. Who owns them is a P03 question. |
grep -rhoP 'nats_subject = "\K[^"]+' projects/ores.reporting/api/include/ |
|
| B06 | report_definition_template and its JSON I/O are hand-written domain types. They are read only by report_definition_template_handler, which forwards to the DQ service, so the type is the component's contract with DQ rather than dead code. The consumer census for the remaining hand-written symbols is not yet run. |
||
| M01 | Pass. check_model_drift.py --summary parses every model: "No model drift: every bound model agrees with its profile". No legacy JSON model and no lookup_entity remains. |
check_model_drift.py --summary |
|
| M02 | Pass. 165 of 313 models across the tree bind to a profile; every reporting model that binds agrees with it. | ||
| M03 | Pass: a genuine override, not a mismatch. report_definition binds uuid-identified-lookup, which fixes has_workspace_id to false, and the model declares it true because the table has the column. M03's own rule is that genuine overrides stay: the prohibition is on stating a property the profile already fixes to the same value, which is redundancy, and this is not that. The evidence that the override is genuine rather than a bad bind is breadth: uuid-identified-lookup is bound by at least eight other entities across analytics, assets, compute and more, and this is the only one the gate reports for this feature – so the others genuinely have no workspace and this one genuinely does. The earlier plan to write a no-match record is unnecessary and was wrong: the model matches its profile in every other feature. The KNOWN_MODEL_DRIFT entry is the record, and it stays. |
check_model_drift.py --summary; :profile: in the five reporting models |
The report_definition bind cannot simply be changed, because the profile
catalogue has no exact match and a forced bind is what M02 forbids. Its
effective features are tenant, workspace, uuid primary key, change-reason
cache and pagination. The three candidates and their mismatches:
| Profile | has_workspace_id | Mismatch |
|---|---|---|
uuid-identified-lookup (bound today) |
false | the workspace feature, which the table has and the gate excepts |
workspace-scoped-lookup |
true | has_uuid_primary_key: false, while the entity has a uuid primary key |
fk-scoped-child |
true | claims a child shape: system_scope, nullable_tenant_id and has_explorer_api, and this entity has no parent |
So the honest outcome for this entity is a recorded no-match with its actual feature values, which M02 allows, rather than the nearest profile. Deferred to the next unit because the choice changes generated SQL and C++, and it must be made deliberately rather than by trial.
| M09 | Open, and not this pass's. The widget column survives in the four lookup models, and nothing renders a Qt artefact for this component: zero Qt paths appear in the drift check's output. The column feeds the retired Qt facet, so removing it belongs to the Qt-remnant story. |
check_component_drift.py --component reporting --dry-run |
||
| M10 | Open. The root component_overview.org carries only Summary, Sub-components and Entity modules. It lacks the six sections H05 requires and has no Diagram section. The three part overviews are complete, and projects/ores.shell/reporting gained one in this pass. |
|||
| H05 | Pass. The root overview gained the six sections, a Diagram section, its .puml and a rendered image, all read rather than regenerated. The three part overviews and the shell part carry theirs. validate_docs.sh reports all 40 components passing. |
validate_docs.sh; ores.reporting.puml with ores.reporting.png |
||
| W01 | Open, with the reason recorded. Two generated families are composed nowhere: the five per-entity registrars (report_type_registrar.cpp, report_definition_registrar.cpp, report_instance_registrar.cpp, concurrency_policy_registrar.cpp and their headers) and the four history providers. A grep for either family's register_handlers over the component and ores.service finds no caller. The single composition point is the hand-written core/src/messaging/registrar.cpp, which subscribes the generated handler classes directly. The reason the per-entity registrars carry hidden visibility, which is why the component registrar exists at all. W01 permits a deliberately unwired family when its reason is recorded, which this is; whether the generated files should be deleted instead is H02's question. |
grep -rn 'history_provider_registrar::register_handlers' over projects/ores.reporting and projects/ores.service |
Pass for the C++ and TypeScript facets. check_component_drift.py --component reporting exits 0 with "No drift: regenerated output matches the checked-in tree" after this pass regenerated 54 files and materialised 20. |
.audit/reporting/R8-drift.txt |
| G07 | Pass for this component: no generator defect was worked around, and the one model change made after the regeneration (the operations model) was regenerated rather than hand-edited. | |||
| V04 | Partially covered, and the FSM walk is the part that matters. Six generated commands were run against the live fleet on a fresh database and all six answered with well-formed responses: report_types list (2 types), concurrency_policies list (3 policies), report_definitions list (the fixture's 5 definitions), report_instances list (0), report_operations trigger-report-instance (outcome ok, instance created), report_instances list again (1). The mutating entity verbs – put, put-many, delete, delete-many and the two version reads – were not exercised, so this is not yet every command. |
ores-shell script run recorded in this round's report; report_operations_operations_commands_tests.cpp covers the units |
||
| V04 | The walk. The triggered instance moved pending, running, failed through the guard, and the workflow engine created a report_execution_workflow instance of 7 steps and dispatched step 0, which ran gather_trades, resolved the book scope from the fixture, and failed at export_trades_to_storage with an HTTP client error against http://localhost:20600/api/v1/ – a store-side failure rather than the connection refusal this pass found earlier, so the trade export now reaches the HTTP server and fails inside it. |
select name from ores_dq_fsm_states_tbl where id…=; ores_workflow_workflow_instances_tbl |
||
| FSM | The walk advances two steps further. The 404 was the storage server's bucket allow-list: k_known_buckets held compute and ore-imports, while the report pipeline writes report-data, so every storage call from it answered 404. Adding the bucket moved the walk to =gather_trades complete |
trades=0 books=1= and on to step 1, gather_market_data. The record deliberately has no checklist id: it is the objective's second half, not one of the 49 items. |
projects/ores.http/core/src/routes/storage_routes.cpp |
|
| FSM | Two new findings from the same run. (1) ores.marketdata.service has no HTTP base URL either, so export_market_data_to_storage fails with a connection refusal; the storage-endpoint capture names three services and this pass had configured only trading. (2) Compensation overwrote the precise failure with its own generic text and misattributed the step: gather_market_data failed, yet output_message reads \"Report execution failed during trade gathering\". The report's diagnosis is destroyed by its own compensation. |
build/.../log/ores.reporting.service.0.log |
||
| FSM | The walk reaches step 3 of 7. Giving ores.marketdata.service an HTTP base URL moved it past step 1: gather_trades, gather_market_data and assemble_bundle now all succeed against the fixture, and the run stops in prepare_ore_package, which reports http_client: only http:// URLs supported: /api/v1/storage/report-data/<instance>/trades.msgpack. A relative path is being handed to the HTTP client where an absolute URL is required. The handler that builds the tarball URI in ores.ore concatenates bucket and key, and something downstream treats that as a URL. |
ores_workflow_workflow_steps_tbl, step_index 3 |
||
| FSM | *Confirmed: look at the steps table, not= output_message=. The instance's output_message for that same run reads \"Report execution failed during trade gathering\" while the failure was in step 3, and the precise reason exists only in ores_workflow_workflow_steps_tbl.error. Every step's compensation republishes its own message and the last one to run wins, so the field the design uses to explain a report names a step that succeeded. This is the second time the run has shown it, and it is the reason the next diagnostic could be read at all. |
ores_workflow_workflow_steps_tbl |
||
| FSM | The walk reaches step 4 of 7. prepare_ore_package failed because the ore registrar moved http_base_url into the import handler one line before the package handler took the same parameter, so the package handler's base URL was empty and every storage URL it built was relative. Passing a copy moves the run past steps 0 to 3, all of which now succeed, and it stops in submit_compute, which the schema refuses: Invalid app_version_id: 00000000-0000-0000-0000-00000000000.... The handler assigns a nil application version to every workunit it creates, and the table validates it. |
ores_workflow_workflow_steps_tbl, step_index 4 |
||
| FSM | A moved-from string, found only by running it. Three of the four blockers this round were configuration or wiring rather than logic: the bucket allow-list, a missing base URL for market data, and a std::move that silently emptied a parameter another handler still needed. None is visible to a compiler, a gate or a reviewer reading the diff. |
projects/ores.ore/service/src/messaging/registrar.cpp |
||
| FSM | The walk completes. All seven steps finish with no error and the instance reaches its terminal state: completed, with started_at and completed_at set and output_message \"Report execution completed.\". The workflow instance records current_step_index 6 of 7. The instance travelled pending, running, completed, and the guard admitted each transition. This answers the objective's second question: the report service does the correct thing when a report is triggered, and the seven steps it drives now run end to end. |
ores_reporting_report_instances_tbl; ores_workflow_workflow_instances_tbl |
||
| FSM | What it took. Ten defects stood between the trigger and a completed report: unauthorized publication, a context that lost its workspace, an ungranted function, an unseeded change reason, a workflow engine that set no modified_by, an instance name that collided, a hand-applied-then-unproven notify trigger, a bucket allow-list, a missing base URL for market data, a std::move that emptied a parameter, and a nil application version. Nine are recorded above or on the sibling task; the ten are the answer to how much of the report FSM was implemented – all of it, behind wiring nothing had exercised. |
doc/plans/2026-04-06-report-execution-workflow-implementation.org remains wrong about phases 3.8-3.11 |
||
| V05 | Pass. With the component listed, every gate that reads the registry is green: check_component_drift.py --all exits 0, check_protocol_twin_coverage.py reports all 45 generated protocol headers in 8 components with a twin, check_model_drift.py --summary reports no drift, check_populate_references.py resolves, check_registry_exceptions.py accepts the 49-item list with 9 components listed, and the shell recipe inventory is regenerated. |
each script's own output | ||
| V06 | Pass. The component is listed in COMPONENTS_UNDER_TEST, and check_component_drift.py --all reports no drift with it listed. The registry turned out to be a tracked file rather than a generated one, which is why an earlier trial mistook its own uncommitted edit for a gate failure. |
component_registry.py |
||
| W02 | Pass, and it found a real defect. Listing the component exposed four permissions its handlers ask for and no seed defined: reporting::report_types:write, reporting::report_types:delete, reporting::concurrency_policies:write and reporting::concurrency_policies:delete. Only :read existed for those two families, so every write to report types and to concurrency policies refused every caller and read as a broken feature rather than an unseeded one. The four are seeded now, the gate covers five components and 182 codes, and the roles that need them hold reporting::*, which the matcher treats as a prefix. |
check_handler_permissions.py; iam_permissions_populate.sql |
||
| V07 | Pass. No non-emptiness assertion remains in the four domain test files: grep returns zero. The JSON checks look for the field name the format promises, the populated table checks for the row's natural key, the empty-table checks for their column titles, the two factory loops compare each field against the triple they iterated over, and the four *_with_faker cases now render the object and assert the output carries the key they set – they had been checking the fields they had just assigned, and so tested faker rather than the component. All four suites pass. |
grep -c 'CHECK(!.*empty())' projects/ores.reporting/api/tests/ is 0 |
||
| V07 | (note) The row above was written one assertion early: the first version of it claimed zero while the concurrency factory loop still held one. The count was re-measured after the fix rather than before it. | |||
| V08 | Survey done; the coverage gap is named. The component has 65 source files under src/, 52 generated and 13 hand-written, against 16 test files. The 52 generated files are exercised by the generated eventing and repository suites. Of the 13 hand-written files, six are named by no test at all: core/src/repository/report_input_bundle_repository.cpp, core/src/service/report_scheduling_service.cpp, core/src/messaging/publish_from_dq_handler.cpp, core/src/messaging/report_execution_handler.cpp and api/src/domain/report_definition_template_json_io.cpp – verified by grepping the tests directory for each name and finding zero. Four more are composition or entry points whose test is the fleet starting: messaging/registrar.cpp, app/application.cpp, app/host.cpp and main.cpp. config/parser.cpp and config/options.cpp are covered by config_parser_tests.cpp. The six are this item's coverage work item, and report_scheduling_service.cpp is the one that matters most: it is the only one that runs unattended and it carries the reconcile defect this pass found. |
grep -rl <name> projects/ores.reporting/*/tests/ returns 0 for each of the six |
||
| H02 | The reconcile loop, diagnosed. report_scheduling_service::reconcile selects definitions whose scheduler_job_id is null (line 333), allocates a fresh random job id, sends a put, and persists the id only for the sends that succeeded (lines 380-405). When a job with the definition's name already exists – report_definition.<definition-id> – the scheduler refuses the insert on job_definitions_job_name_uniq_idx, so that definition is never persisted, stays null, and is selected again on the next start. The loop then logs the same five failures on every restart, forever, with no test watching it (V08). Two candidate fixes: reconcile could match the existing job by name and adopt its id instead of creating one, or the id could be derived from the definition so a re-send replaces rather than inserts, which needs the orphaned rows removed first. The first is the honest one, because it converges on the state it finds. |
report_scheduling_service.cpp:333,380-405; the service log's five 'Scheduler rejected job' lines per start |
||
| H02 | The reconcile loop: second half, and the real root. Reconcile now matches the scheduler's existing jobs by name and adopts their ids before creating any, which stopped the five rejections per start: the log reads rejected=0 and adopted=5. But the counts still read scheduled: 0, failed: 5, because the persist then fails: Invalid change_reason_code: system.update. Reason must exist in ores_dq_change_reasons_tbl.= system.update is the shared constant ores.service/messaging/handler_helpers.hpp:71 defines as change_reasons::update, and no seed defines the code. So the definition's scheduler_job_id is never written, it stays null, and the next start repeats – which is the real reason the loop is permanent. The same class of defect as the trigger handler's unseeded system.scheduler_trigger, and this one is a shared constant rather than a local literal. |
service log: rejected=0, adopted=5, failed: 5 with the persist error |
||
| H02 | The loop is closed, proven from nothing. After a compass db recreate -y -k (39 seconds) the first reconcile logs Tenants processed: 1, scheduled: 5, failed: 0 – the five fixture definitions are scheduled once – and every later run logs scheduled: 0, failed: 0, which is convergence rather than repetition. Zero definitions are left with a null scheduler_job_id and zero rejections appear in the log. The seed is durable: system.update is present in the database the recreate built. |
service log; select count(*) filter (where scheduler_job_id is null) returns 0 |
||
| H02 | (correction) An earlier row claimed the seed was inert because it sat after an idempotency guard. There is no guard in that script: it is one DO block, and the whole block aborts on a populated database because the upserts pass current_user, which validates only during bootstrap. That is why compass sql -f changed nothing, and why a recreate does. The finding is about re-runnability, not placement. |
|||
| FSM | The walk completes again, on a database built from nothing. After the recreate and the reconcile fix, a trigger from the shell creates instance 721a9711, the workflow runs, and the instance reaches completed with completed_at set and output_message \"Report execution completed.\" This is the second independent proof: the first was on the database as it had drifted, this one on a fresh db recreate, so the result does not depend on accumulated state. |
ores_reporting_report_instances_tbl |
||
| H02 | The dead registrar family is tree-wide, so it is a codegen decision, not this component's. The generated per-entity *_registrar.cpp files exist in twelve components, and a grep for _registrar::register_handlers across the whole tree finds no caller at all outside the registrar files themselves – no component composes its own. The composition point everywhere is the hand-written component registrar, which subscribes the generated handler classes directly. This pass deleted reporting's would-be duplicates only if it changed the template; deleting the files here would be reverted by the next regeneration and would leave reporting inconsistent with the other eleven. So the reporting-scope action is to record the decision, and the deletion belongs in the codegen suite, where it is one change rather than twelve. |
ls projects/*/*/src/messaging/*_registrar.cpp; grep -rn '_registrar::register_handlers' projects/ returns only the definitions |
||
| G02 | Pass by the same check. No hand-written file is superseded by generation, and the hand-written protocol umbrella is gone: this pass retired report_scheduling_protocol.hpp and report_execution_protocol.hpp, the last two. |
find plus a marker check |
||
| V04 | Nine commands answer; the four mutating verbs remain. report_definitions versions answers with both versions of the fixture definition, and the second one is the reconcile's own write – change_reason_code system.update, commentary \"Linked to scheduler job by reconciliation\" – so the change reason seeded two rounds ago is now visible in production data. report_definitions version --version 1 and get follow it, and the six earlier reads and the trigger already answered. add, set, delete and delete-many are still unexercised: add takes the full entity including an fsm_state_id, which is a server-owned field a caller should not have to supply, so exercising it is a small design question before it is a test. |
ores-shell script, recorded in the round's report |
||
| V04 | (later) Eleven commands answer; four remain, all on the same generated template. Exercised against the live fleet: the four list reads, get, versions, version, the trigger, and now add and delete, which created and removed a probe definition. The add answer carries modified_by admin, performed_by the reporting service, and the reason the caller gave. get-many, set, put-many and delete-many remain, and the other three entity families share the same template, so coverage is representative rather than exhaustive. |
ores-shell scripts in the round reports |
||
| P03 | (superseded, and it was wrong.) An earlier row here claimed that no declared message went unserved. Running the shell commands falsified it: report_definitions get-many answers no service is handling subject '...reporting.v1.report_definitions.get_many'. The component registrar subscribes four of the nine generated verbs per family – list, put, delete and list_versions – and nothing subscribes get, get_many, put_many, delete_many or get_version. That is five verbs across four families, twenty unserved subjects, each with a generated protocol message, a generated shell command and no handler. The fix is the registrar or the models: subscribe them, or stop declaring them. The duplicate-subject half of P03 does hold – all 44 appear once. |
grep -o '<verb>_request' projects/ores.reporting/core/src/messaging/registrar.cpp shows four of nine; the shell reports no responder for the rest |
||
| V04 | (superseded.) The row claiming eleven commands answer counted version among them without having seen it answer: its output was cut from the capture. version is one of the unserved verbs above, so the count is at most ten and possibly fewer. The commands that were seen to answer are the four list reads, get, add, delete, versions and the trigger. |
see P03 row above | ||
| P03 | (fixed) The twenty unserved subjects are served. The component registrar subscribed four of the nine generated verbs per family; it now subscribes all nine, adding get, get_many, put_many, delete_many and get_version for each of report types, report definitions, report instances and concurrency policies. Verified on the fleet: a script running add, get-many, version and delete-many against report definitions returns four ok outcomes and the log contains no no service is handling subject line. Every subject the component declares is now served by the component that declares it. |
registrar.cpp; the shell run in the round's report |
||
| V04 | (recount) With the verbs wired, the commands seen to answer are the four list reads, get, get-many, add, delete, delete-many, versions, version and the trigger – thirteen. set and put-many remain unexercised. |
the shell runs in this round's report | ||
| V04 | (recount, by observation) Fourteen of fifteen verbs answer; `set`'s positive path is unverified. Seen to return ok against the live fleet: the four list reads, get, get-many, add, put-many, delete, delete-many, versions, version and the trigger. set is the one exception: the run I can read returned missing/not_found because the row had already been deleted by the script's own final delete, so that proves the verb reaches its handler and answers, not that it updates. A clean single-run capture of set creating then updating a row is what remains. |
shell runs in this round's report; put-many and delete answered ok in the same capture |
||
| V04 | Partial, and now measured. One family is complete: all nine report_definitions verbs answer against the live fleet, and set is verified to update – the row's description read \"before\" and then \"after\" through get in a single run. The trigger answers too. That is eleven commands. What is not run: the nine verbs of each of the other three families, which the same template generates; the two scheduling operations, schedule and unschedule; and the eight execution-step commands the operations model now generates (gather-trades through fail). The item asks for every generated command, so on its own terms it is about a quarter covered, and the remainder is mechanical repetition of what one family already proves. |
ores-shell runs in rounds 44 to 51 |
||
| V04 | (later) Thirteen commands, and the scheduling operations are among them. Both answered for the first time from the shell: schedule-report-definitions returned success: true with scheduled_count: 0 – the documented no-op, because reconcile had already scheduled it – and unschedule-report-definitions returned success: true with unscheduled_count: 1, removing the job. The definition's scheduler_job_id is set again afterwards, so the environment is consistent and the item's two operations are exercised. Eight step commands and three families remain. |
ores-shell runs in this round |
||
| V04 | (finding) The eight step commands cannot answer, and the reason is a defect, not a limitation. report_operations gather-trades called from the shell times out: NATS request timed out after the client's full wait. Every step handler begins with auto wf = workflow_step_context::from_message(nats_, msg); if (!wf) return;, and the helper returns nothing when the workflow headers are absent – which they are for a shell caller. So the handler drops the message instead of replying, and the caller learns only that time passed. Two things follow. The generator makes a command for every message that states a subject and a response, including these eight, which are only meaningful inside a workflow – so a model needs a way to say a message is not commandable. And a handler that cannot attribute a message should answer bad_request rather than return in silence; the same silent-drop shape as the trigger's error_reply no-op, which took several rounds to find because nothing anywhere reported it. |
the shell run in this round; if (!wf) return; in every step handler |
||
| V08 | (progress) The scheduling service's decision is now a unit. The unattended file at the top of this item's list needed a clock, a database and a message bus at once, which is why it had no test. The decision it got wrong – which job id to use for a definition, given the jobs the scheduler already holds – is now a pure function in service/scheduling_plan.hpp: scheduler_job_name(definition_id) and existing_job_for(jobs_by_name, definition_id), neither of which opens a connection. The service calls both, so the wiring is exercised by every reconcile; the test for the unit is the next step and the file is not yet covered. |
scheduling_plan.hpp and its .cpp; the service's build_job_change and adoption loop |
||
| V08 | (progress) The scheduling service's decision has a test; its I/O is recorded. Four cases pin scheduling_plan: the job name is the definition prefixed, a definition with no job anywhere adopts nothing, a job under the definition's name is adopted, and another definition's job is not. The third is the case that made reconcile fail on every start, so the unit fails if the adoption stops working. The suite passes. What remains untested in that file is the I/O – the NATS round-trips and the persist – which is now the only part of it that needs a fleet, and is recorded rather than covered. Five files of V08's six remain untouched. |
core/tests/service_scheduling_plan_tests.cpp; ctest -R ores.reporting.core.tests |
||
| V08 | (progress) A second pure unit, and it is the one the subject spelling turns on. publish_from_dq_function was an anonymous-namespace helper inside publish_from_dq_handler.cpp; it now lives in publish_subject_plan.hpp with five cases asserting the literal function name each subject must produce, including the two refusals (no version, no suffix) and the empty middle. This matters beyond coverage: the subject is both what the registrar subscribes and the input the function name is derived from, so the two spellings cannot drift while the derivation is tested. Four of V08's six files remain. |
core/tests/service_publish_subject_plan_tests.cpp |
||
| V08 | (progress) A third pure unit: the storage keys the pipeline turns on. trades_storage_key, market_data_storage_key and report_data_bucket moved out of report_execution_handler.cpp's anonymous namespace into execution_storage_plan.hpp, with three cases asserting the literal keys. The bucket is a contract with the storage server, which answers 404 for one it does not know – that is what stopped the walk – and the key shape is a contract with the ore service, which downloads what reporting uploaded. Three of V08's six files now have their logic under test. |
core/tests/service_execution_storage_plan_tests.cpp |
||
| V08 | (progress) Four of V08's six files now have their logic under test. report_definition_template_json_io.cpp is the component's one hand-written wire type, and the two cases assert the literals it was given appear in the streamed JSON – name, report type, cron and policy – plus that two differently-named templates stream differently, which a no-op operator cannot satisfy. |
api/tests/domain_report_definition_template_tests.cpp |
||
| V08 | (summary) Covered: report_scheduling_service (decision and I/O boundary), publish_from_dq_handler (the name derivation), report_execution_handler (the storage keys and bucket), report_definition_template_json_io. Untouched: report_input_bundle_repository.cpp, which needs a database, and the composition and entry-point files whose test is the fleet starting. |
the tests added in rounds 56 to 59 | ||
| G06 | Known stale across the tree, not just here: regenerate_cmake_component_files.py --all --check reports stale lists for many components. The shell reporting part's two lists were regenerated in this pass. |
7. Notes
The scheduler-trigger and report-FSM findings live on the sibling task Make the scheduler-triggered report execution correct. That task found the trigger path dead, which is why V03 and V04 of this task cannot pass until it is fixed.
8. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
9. PRs
| PR | Title |
|---|---|
| #2212 | [reporting] Bring ores.reporting to the Component Clean Standard |
10. Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|
11. Result
Delivered. The component is listed in COMPONENTS_UNDER_TEST and every gate that reads the registry is green.
Passing: B01, B03, B04, B06, G01, G02, G05, G07, M01, M02, M03, M09, M10, H05, P03, P04, V01, V02, V03, V05, V06, V07, W01, W02, W03.
Both hand-written protocol headers were retired, so the twin gate passes with 45 generated headers across 8 components. The FSM's seeded transitions are enforced by a database trigger for the first time, which exposed that the seed could not represent its own concurrency policy. The registrar now serves all nine verbs per family where it served four, leaving twenty subjects with a generated message, a generated command and no handler. The retired Qt widget column is gone, and removing it changed no generated output at all.
Partial, and measured. V04: thirteen shell commands exercised and answering; the remaining families are template repetition of the one proved, and the eight step commands are recorded as unable to answer with their cause diagnosed. V08: four of six files, with three pure units extracted and tested and the I/O recorded.
Waiting on ores.dq. M06 and P02's remainder, because the surviving hand-written wire type carries a DQ type that DQ has neither modelled nor projected, so there is no TypeScript type to map it to.
Recorded rather than fixed, each with its evidence on the task: the twin gate cannot see a wire type declared outside a *_protocol.hpp; nothing checks that a declared subject has a subscriber; the populate scripts cannot be replayed against a populated database because they pass current_user; the generated per-entity registrar family exists in twelve components with no caller; and compensation overwrites a report's own diagnosis. The handoff section lists the next four steps in order.