Task: Write up the post-merge system-test report for the compute and iam binds

Table of Contents

This page documents a task in the Manage the compute grid from the shell story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

The ores.compute bind (PR #2012) and the ores.iam bind (PR #2014) regenerate whole per-entity messaging, presentation and SQL files from the org models. The checks that gate a merge cannot see behaviour the templates never knew: ctest runs as privileged roles, the drift checks measure template fidelity, and no comparison of pre-bind vs post-bind behaviour runs at merge time. The desktop UI was unavailable, so by directive the post-merge verification ran through ores.shell only. This task records that system test as the #79 report: #79 is the test program's harness task number, not a GitHub reference. The report covers the findings (F1-F5) with their resolution state, the smoke evidence and residue, the root-cause analysis, the limitations of a shell-only run, and the follow-ups it generated.

2. Status

Field Value
State DONE
Parent story Manage the compute grid from the shell
Now Nothing.
Waiting on Nothing.
Next The follow-ups the report records: the UI-gap story, the F2/F3 fix PRs, the GitHub Actions hotfix.
Last touched 2026-09-06

3. Acceptance

  • Every finding the system test produced is recorded with its resolution state and its owning task or PR.
  • The smoke runs (smoke-fix1/2/3) and their residue are accounted for.
  • The root-cause analysis states why the merge-time checks could not catch the dropped behaviour.
  • The shell-only limitations and the UI-gap story that addresses them are recorded.
  • Every follow-up names its home: an open task, a story, or a backlog item.

4. Plan

The system test ran against brave_hopper after the bind merges, in three legs, all driven from ores.shell:

  1. S1, the iam/tenants leg, on a freshly recreated and fully seeded database: tenant provisioning, lookup, RLS isolation, delete rules, the service-user grant surface, and the account_party junction path. This leg produced F1, F2 and F3.
  2. S2, the compute leg, through this story's shell verbs and its smoke script: batch dispatch, drain, result submit, host exercise. This leg produced F4 and F5.
  3. Re-verification legs on the fix branches: fresh-database tenant provisioning under the real service-user path for F1 (PR #2016), and fleet restart plus smoke script for F4/F5 (PR #2015).

Each leg ran against the fleet rebuilt and restarted from the merged tree. Evidence came from the shell and from the services' logs. No UI session was available; the flows the shell cannot drive are tracked in a separate story.

5. Notes

The system-test program and its two fix PRs predate this task. The record spread across harness tasks, PR bodies and review rounds; this task consolidates it into one report document. The fix tasks link back here for their residue accounting: Fix the compute grid dispatch and wrapper result-submit channels dropped by the ores.compute bind and Grant iam_service select on refdata for the account_party soft FK.

6. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario, run through the QA Validation Runner panel) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

7. PRs

PR Title
#2018 [agile] Post-merge system-test report for the compute and iam binds

8. Review

Comment summary File Decision Notes
       

9. Result

9.1. Findings

# Finding Class Status Owning task / PR
F1 Tenant provisioning on a fresh database failed: permission denied for table ores_refdata_parties_tbl. The bind regeneration of the account_party junction dropped the function's legacy SECURITY DEFINER attribute; under the plain invoker idiom the soft-FK validation subselect against ores_refdata_parties_tbl runs as the iam service user, who held no select grant (the service registry granted ores_assets_ and ores_dq_ only). Runtime privilege context changed by template modernisation FIXED, merged Grant iam_service select on refdata, PR #2016
F2 tenants add fails end to end: the shell sends the bare reason code new_record while the strict catalogue requires the namespaced system.new_record. One stale literal in tenants_commands.cpp:152, dated 2026-03-15; the provisioning handler and the change_reason_constants header have used the namespaced form since the 2026-05-18 seeds. Masked on old databases: an empty reasons catalogue made the validation pass through. Stale literal, fresh-DB-exposed OPEN Own fix PR pending
F3 Deleting the system tenant succeeds. The recipe prose documents "The system tenant cannot be deleted", but no guard exists in the service: the handler checks only the iam::tenants:delete permission and the SQL delete rule soft-terminates any row, the system tenant included. The id/code check constraint is insert-only. The protection is UI-only. Guard present in one client only OPEN Own fix PR pending
F4 The bind dropped the grid dispatch side effect. Pre-bind, workunit_handler::save created the result row and published the JetStream work_assignment_event to compute.v1.work.assignments.{tenant}.{platform_code}. Post-bind, save writes the workunit row only: a live dispatch of 10 workunits fired 10 workunit_changed events and published nothing; zero result rows existed and the wrapper fleet sat idle on its subscription. Side effect inside a surviving subject FIXED, merged Fix the compute grid dispatch, PR #2015
F5 The bind dropped the trusted wrapper result-submit channel. The NATS subject-surface audit (23 subjects pre-bind, 33 post-bind) found exactly one endpoint with no successor: compute.v1.results.submit. Wrappers hold no user JWT, so the generated JWT-gated save_result CRUD rejected every submission. Dropped endpoint, visible to a subject diff FIXED, merged Fix the compute grid dispatch, PR #2015

The audit behind F5 is the "nothing else was dropped" proof. Every other pre-bind subject survives or grows: hosts, apps, batches, workunits and results gain delete/history/list_by_* supersets, the app-versions hyphen set is a pure rename to app_versions with functional equivalents, and the hand-written domains (platforms, telemetry, work.heartbeat/pull/assignments/reap) are byte-identical on both sides. F4 shows the limit of that audit: the dispatch loss hid inside a surviving subject, as a side effect of save. Only a live smoke can see that class.

9.2. Smoke evidence and residue

  • Smoke-fix1 ran against the broken dispatch: 10 workunits saved, no results. Its ten state 2 rows are inert pre-fix submit failures; the reaper re-queues state 4 only, so they cannot resurrect. The st250905 batch (10 workunits, zero results) was dispatched in the same broken-dispatch phase, so no clean pre-merge smoke baseline exists in the database.
  • Smoke-fix2 hit 3/5 hosts. After a fleet restart the JetStream queue group needs a settle period before it covers every member; the distribution skew was transient.
  • Smoke-fix3 ran after the F4/F5 restore: SMOKE PASS, 10/10 results Success, 5/5 online hosts exercised, canonical_result_id set on every result, batch closed by the Assimilator.
  • One stale queue consumer from the pre-fix fleet lingered after restart; the fixed fleet consumes on the restored subject.

9.3. Root-cause analysis

The bind PRs replaced hand-written per-entity files with generated ones from the org models (the compute bind alone changed 202 files, deleted none). The templates know the entity CRUD surface; they do not know side effects the grid depends on, and a bind rewrites whole files from the model. Whatever the model does not state is absent from the new file, and no before-vs-after semantics comparison runs at bind time.

The existing gates cannot see that class of loss:

  • The drift check measures template fidelity: regenerate from the models and diff the tree. A bind commits exactly the regeneration output, so the tree equals the regen by construction and the check passes. It cannot see behaviour the template never knew.
  • The cmake-sources drift check tracks file lists, not semantics.
  • ctest covers the generated surface while running as privileged roles. It cannot see a dropped side effect, and it cannot see a missing service-user grant.
  • No check compares pre-bind behaviour to post-bind behaviour. The subject-surface audit (F5) is that comparison for endpoints; the live smoke (F4) is that comparison for side effects.

The F1 class adds a database dimension: some regressions appear only on a freshly recreated, fully seeded database. Old databases carried manual grants and empty catalogues, so both F1 and F2 passed there and failed on the fresh one.

9.4. Limitations

  • No UI driving. Every flow was exercised through ores.shell, per the shell-only testing directive; the UI-only test gaps are tracked in a separate story.
  • Redundancy above 1 was not exercised live; the smoke grid ran at redundancy 1 on a single platform (x64-linux).
  • Single-instance topology only. The Validator threshold and the dispatch top-up branch therefore have no live evidence.
  • ctest runs as privileged roles and cannot exercise the service-user grant path (F1) or the wrapper submit path (F5); the live runs are the only evidence for those.

9.5. Follow-ups

  • UI-gap story: a separate sprint_25 story covers UI verification of the compute-grid and iam/tenant flows this report tested shell-only, to run when a UI session is available.
  • Tenants findings F2 and F3 need their own fix PRs; both are C++ or SQL service changes with full build and ctest verification.
  • Scale-out prerequisite, recorded from the PR #2015 review: the canonical-assignment guarded update and dispatch idempotency across instances must be designed together once the service runs more than one instance.
  • Test-coverage follow-up from the PR #2015 review: the dispatch top-up branch and the redundancy-above-1 Validator path have no unit coverage. Both handlers construct their repositories and NATS client directly; unit tests need a small dependency-injection seam first.
  • Codegen banners: every codegen-emitted file must carry a generated banner (task in the entity-classification-drift-rollout story). Banners are necessary legibility, not proof of lossless regeneration.
  • Testing section on PR create: compass pr create gains an always-rendered Testing section (plan, what was tested, limitations) so every PR records its verification.
  • GitHub Actions health: main's continuous builds (Linux, MacOS, Windows) and the nightly builds are red as of 2026-09-06, across all OSs. The fix is a hotfix in flight, and an occasional GitHub Actions health check joins the program flow so a red main is noticed between merge cycles.

Status at close: Report written. Findings F1/F4/F5 closed by PRs #2016/#2015; F2 and F3 await their own fix PRs.

Emacs 29.3 (Org mode 9.6.15)