Task: Add a compute grid smoke-test script and prove it live
Table of Contents
This page documents a task in the Manage the compute grid from the shell story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
A single library script exercises the whole compute grid from the
shell: it adds a batch, dispatches 10 to 20 consecutive jobs, watches
compute grid-stats --watch --smoke until the batch drains, and
asserts every online host was exercised and every result has outcome
Success. The script is the grid's end-to-end health check.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Manage the compute grid from the shell |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-08-25 |
3. Acceptance
- One library script drives the whole lifecycle: add a batch with
--smoke(count bounded to 10 to 20), dispatch the jobs to a target app version, watchgrid-stats --watch --smokeuntil the batch drains. - The script fails if any result does not have outcome Success.
- The script fails if any host online at the drain transition was not exercised (has no result in the batch).
- The script is generated from a recipe (tangle) and proven live against a serving environment.
4. Plan
The script is generated from the recipe
how_do_i_smoke_test_the_compute_grid_from_the_shell by the standard
tangle. Its ores-shell block chains the verbs: add-batch
smoke-$SMOKE_REF 15 --smoke, dispatch-batch smoke-$SMOKE_REF 15
$APP_VERSION_ID $INPUT_TARBALL, grid-stats --watch smoke-$SMOKE_REF
--smoke. The job count is 15; the --smoke switches bound it to the
10 to 20 range and turn on the assert machinery. The assert lives in
the grid-stats verb (see the monitoring task): outcome Success on
every result, and every online host with at least one result in the
batch.
5. Notes
- Live proof (2026-08-11): the full sequence ran against brave_hopper
with hello 1.0.0. The 10-job batch drained in
2 seconds with all results Success; the smoke assert then correctly FAILED on the phantom host ~clever-echidna(online, zero results in the batch) and the script aborted as designed. This proves the assert machinery end-to-end; the acceptance cannot go fully green until the phantom driver is stopped or its host removed (see the monitoring task notes for the diagnosis: remote driver on 192.168.1.22, _INBOX-only, never pulls work). - The environment carries a second known defect: the node telemetry codec mismatch (msgpack publish vs JSON decode) leaves the node_samples table empty, so the grid-stats node rows are empty — the summary and smoke asserts are unaffected.
- Recipe and script live in
doc/recipes/shell/compute/andprojects/ores.shell/scripts/library/compute/, indexed under theComputesection ofdoc/recipes/shell/shell.org.
6. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario, run
through the QA Validation Runner panel) that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
7. PRs
| PR | Title |
|---|---|
| #1992 | [ores.shell] Add compute grid management verbs to the shell |
8. Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|
9. Result
- The library script (tangled from
how_do_i_smoke_test_the_compute_grid_from_the_shell) drives the whole grid end-to-end:add-batch --smoke(count bounded to 10 to 20),dispatch-batch, thengrid-stats --watch --smokeuntil the batch drains. - Proven live against brave_hopper (2026-08-11): the 10-job batch drained in ~2 seconds with all results Success; the smoke assert then correctly FAILED on the phantom host (online, zero results in the batch) and the script aborted as designed — the assert machinery proven end-to-end.
- A fully green run requires the phantom driver host to stop
heartbeating; the node telemetry codec defect is recorded with the
monitoring task. Recipe and script live in
doc/recipes/shell/compute/andprojects/ores.shell/scripts/library/compute/, indexed under theComputesection ofdoc/recipes/shell/shell.org.