Task: Add a compute grid smoke-test script and prove it live

Table of Contents

This page documents a task in the Manage the compute grid from the shell story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

A single library script exercises the whole compute grid from the shell: it adds a batch, dispatches 10 to 20 consecutive jobs, watches compute grid-stats --watch --smoke until the batch drains, and asserts every online host was exercised and every result has outcome Success. The script is the grid's end-to-end health check.

2. Status

Field Value
State DONE
Parent story Manage the compute grid from the shell
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-08-25

3. Acceptance

  • One library script drives the whole lifecycle: add a batch with --smoke (count bounded to 10 to 20), dispatch the jobs to a target app version, watch grid-stats --watch --smoke until the batch drains.
  • The script fails if any result does not have outcome Success.
  • The script fails if any host online at the drain transition was not exercised (has no result in the batch).
  • The script is generated from a recipe (tangle) and proven live against a serving environment.

4. Plan

The script is generated from the recipe how_do_i_smoke_test_the_compute_grid_from_the_shell by the standard tangle. Its ores-shell block chains the verbs: add-batch smoke-$SMOKE_REF 15 --smoke, dispatch-batch smoke-$SMOKE_REF 15 $APP_VERSION_ID $INPUT_TARBALL, grid-stats --watch smoke-$SMOKE_REF --smoke. The job count is 15; the --smoke switches bound it to the 10 to 20 range and turn on the assert machinery. The assert lives in the grid-stats verb (see the monitoring task): outcome Success on every result, and every online host with at least one result in the batch.

5. Notes

  • Live proof (2026-08-11): the full sequence ran against brave_hopper with hello 1.0.0. The 10-job batch drained in 2 seconds with all results Success; the smoke assert then correctly FAILED on the phantom host ~clever-echidna (online, zero results in the batch) and the script aborted as designed. This proves the assert machinery end-to-end; the acceptance cannot go fully green until the phantom driver is stopped or its host removed (see the monitoring task notes for the diagnosis: remote driver on 192.168.1.22, _INBOX-only, never pulls work).
  • The environment carries a second known defect: the node telemetry codec mismatch (msgpack publish vs JSON decode) leaves the node_samples table empty, so the grid-stats node rows are empty — the summary and smoke asserts are unaffected.
  • Recipe and script live in doc/recipes/shell/compute/ and projects/ores.shell/scripts/library/compute/, indexed under the Compute section of doc/recipes/shell/shell.org.

6. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario, run through the QA Validation Runner panel) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

7. PRs

PR Title
#1992 [ores.shell] Add compute grid management verbs to the shell

8. Review

Comment summary File Decision Notes
       

9. Result

  • The library script (tangled from how_do_i_smoke_test_the_compute_grid_from_the_shell) drives the whole grid end-to-end: add-batch --smoke (count bounded to 10 to 20), dispatch-batch, then grid-stats --watch --smoke until the batch drains.
  • Proven live against brave_hopper (2026-08-11): the 10-job batch drained in ~2 seconds with all results Success; the smoke assert then correctly FAILED on the phantom host (online, zero results in the batch) and the script aborted as designed — the assert machinery proven end-to-end.
  • A fully green run requires the phantom driver host to stop heartbeating; the node telemetry codec defect is recorded with the monitoring task. Recipe and script live in doc/recipes/shell/compute/ and projects/ores.shell/scripts/library/compute/, indexed under the Compute section of doc/recipes/shell/shell.org.

Emacs 29.3 (Org mode 9.6.15)