Story: QA Validation Runner: in-app test tracking for the multi-worktree fleet

Table of Contents

This page documents a story in Sprint 22. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

Goal

Second story of the QA Validation Runner epic (see ores.orgmode: C++ org-mode parsing and org-roam link resolution for the first — this story depends on it and does no org parsing of its own).

The fleet has grown to 10 worktrees, most of them wanting UI-level verification of their in-flight story or task, but there is only one human tester. In practice the tester loses track of what is being tested, on which branch/environment, and what was found — verification notes end up scattered across chat, or not recorded at all.

Add a QA Validation Runner to ores.qt: a dockable panel that turns a scenario (a checklist of manual steps tied to a story/task) into a guided, trackable test session, and writes the result back next to the docs that drove it — not to a JSON file nobody reads again.

Gemini's initial analysis (doc/analysis/gemini_test_ui.org, mockup doc/analysis/gemini_test_ui_mockup.png) proposes a test_scenario.json / test_result.json protocol so an AI worker can hand a checklist to the UI and consume the result back. The panel UX (docked checklist, PASS/FAIL toggle, freeform notes, "launch target dialog") is a good starting point, but the storage format changes direction for this codebase: the scenario is a new org-mode document type, not JSON.

  • A new doc type — provisionally test_scenario — is scaffolded the same way a task or story is: compass add test_scenario, saved under the driving story's directory, linked (id: link both ways) to the task it verifies. It is filled in with target dialog, title, and the checklist of steps before testing starts — an ordinary org file, reviewable in a PR, greppable, renderable on the site like any other doc.
  • The Qt panel reads/writes this doc via the ores.orgmode component from the prerequisite story — it does not implement its own org parsing. What this story adds on top: rewriting the * Results section in place on save (a targeted write, not something ores.orgmode's reader needs to support generically), and the UI itself.
  • Workflow: instantiate the scenario doc under the story, link it to the task, fill in the details — then point the Qt panel at that file. The tester runs through the checklist in the app, ticks steps, writes notes, sets PASS/FAIL, hits save. The panel rewrites the scenario doc's * Results section in place (step outcomes, notes, status, environment metadata — branch/commit/worktree identity read from the checkout, not hand-entered) and stamps the compass journal. The doc is now the single source of truth: git diff shows the result, the website renders it, and the next Claude session can just read the file.

A second, closely-related gap: with this much parallel work across the fleet, the tester (and Claude, picking a session back up) loses the context of what a given scenario is even testing — the story/task doc has that context, but nobody wants to leave the running app and go digging through the site or the org file by hand mid-test. So the same panel (or a sibling tab) should also render the driving story and its tasks — title, goal, acceptance, status, tasks table — using the ores.orgmode document structure to drive a simple, structure-aware renderer (headings, lists, tables, basic emphasis is enough; not a faithful org renderer).

The tester also needs a fleet-level view: which worktree is asking for what test, so they can pick one scenario at a time instead of holding it all in their head. Compass already renders this info (compass fleet, compass show <task>) — this story should establish how the Qt panel surfaces it without duplicating compass's data model.

Status

Field Value
State DONE
Parent sprint Sprint 22
Now Nothing.
Waiting on Nothing.
Next None.
Last touched 2026-07-10

Acceptance

  • [ ] A new test_scenario org doc type is defined (frontmatter, required sections) and scaffoldable via compass add test_scenario, saved under the driving story's directory and id:-linked to the task it verifies.
  • [ ] ores.qt consumes the prerequisite ores.orgmode component to parse test_scenario and story/task docs — no org parsing reimplemented in Qt. The panel adds only the * Results-section rewrite-on-save and a simple structure-aware renderer for display (headings, lists, tables, basic emphasis).
  • [ ] ores.qt gains a dockable QA Validation Runner panel (debug builds only) that opens a test_scenario doc, shows its info (title, target dialog, steps), lets the tester tick steps, capture freeform notes, and set a PASS/FAIL outcome.
  • [ ] Environment metadata (branch, commit, worktree identity) is read from the running checkout, not hand-entered by the tester.
  • [ ] Submitting a result rewrites the scenario doc's * Results section in place (step outcomes, notes, status, environment metadata) — no standalone JSON artefact — and stamps the compass journal.
  • [ ] The same panel (or a sibling tab) can render the driving story/task docs (goal, acceptance, status, tasks table) so the tester doesn't need to leave the app to recover context.
  • [ ] ~The tester can see, from within the panel or a companion list, which worktrees/tasks in the fleet currently have a scenario waiting~ — abandoned (see Surface pending scenarios across the fleet): no fleet-coordination source of truth exists yet to build this on; revisit once one does.
  • [ ] Rendered test_scenario docs (with their filled-in * Results) show up correctly on the published site, same as any other doc.
  • [ ] Works end-to-end for at least one real story from this sprint as a pilot before being declared done.

Tasks

Task State Start End Description
Define the test_scenario org doc type DONE 2026-07-07 2026-07-07 Define and scaffold a new test_scenario org doc type (frontmatter, Scenario Info/Steps/Results sections), following the same doc-type-registration path as story/task, so it's creatable, indexable, and site-renderable like any other agile doc.
Wire ores.qt to ores.orgmode and add the Results-rewrite + doc renderer DONE 2026-07-08 2026-07-08 Add ores.orgmode as a Qt dependency; implement the targeted * Results section rewrite-on-save for test_scenario docs, and a simple structure-aware renderer (headings/lists/tables/basic emphasis) shared by the scenario panel and the story/task viewer.
Build the QA Validation Runner dockable panel DONE 2026-07-08 2026-07-08 Qt dockable panel (debug builds only): open a test_scenario doc, show its info and checklist, let the tester tick steps/write notes/set PASS-FAIL, and save (rewriting the doc's Results section and stamping the compass journal).
Add the story/task context viewer DONE 2026-07-08 2026-07-08 Render the driving story/task docs (goal, acceptance, status, tasks table) in the same panel or a sibling tab, using the shared structure-aware renderer, so the tester doesn't need to leave the app to recover context.
Surface pending scenarios across the fleet ABANDONED 2026-07-08 2026-07-08 Show which worktrees/tasks in the fleet currently have a scenario waiting, sourced from compass's existing fleet/task data (not duplicated) — a companion list or panel section. Abandoned: no proper fleet coordination exists yet to build this on.
Pilot end-to-end on a real sprint story DONE   2026-07-10 Piloted on the Currency Pairs manual chapter screenshot scenario: 5 steps run, Results written back, journal stamped, site rendered correctly.
Add the qa_validation_runner_enabled system setting DONE 2026-07-08 2026-07-08 Seed system.qa_validation_runner_enabled as a boolean system setting so the QA Validation Runner's visibility is a runtime, operator-controlled decision, not a compile-time macro.
Add screenshot capture to the QA Validation Runner panel DONE 2026-07-09 2026-07-10 Add a 'Screenshot' toolbar action next to Open/Save: grab the full primary screen via QScreen::grabWindow(0), save as PNG next to the open scenario doc (same directory as scenarioPath_), named after the scenario slug/step/timestamp. Region-select capture (QRubberBand over a full-screen grab, drag-select, crop, save) as a documented fast-follow, not required for the first cut.

Decisions

  • Scenario and result storage is org-mode docs under the story, not JSON files as Gemini's analysis proposed. Rationale: this project already treats org docs as the source of truth for everything agile — git history, PR review, compass indexing, site rendering all come for free; a JSON sidecar would be a second, disconnected source of truth nobody browses after the fact.
  • Org reading/writing is not reimplemented in ores.qt — it consumes the ores.orgmode component from the prerequisite story in this epic, so the parsing investment is made once and shared with ores.shell and anything else that needs it.
  • The panel lives in ores.qt.admin's AdminPlugin (contributing System > Testing via setup_menus()), not hardcoded into MainWindow — it's an administrative/testing tool, the same kind of feature as System > Configuration=/=Administration.
  • Visibility is a runtime system setting (system.qa_validation_runner_enabled), not a compile-time #ifdef: a fleet-wide manual-testing tool shouldn't be stripped from release builds, and whether it's available is an operator/deployment decision. Required widening shared_menus_context with main_window=/=mdi_area=/=client_manager (already constructed before setup_menus() runs) since this is the first plugin feature that needs to work before login — narrowing the pre-login gate for that surfaced a separate pre-existing gap (System > Configuration=/=Administration were never independently login-gated, only protected by the old blanket System-menu disable), fixed alongside it.
  • Each scenario step is its own org heading (not a bullet checklist) with its own *** Result child heading, not one aggregate results table — steps need per-item PASS/FAIL/PENDING/notes, and a heading gives each step a natural place to carry both its instructions and its own result. The panel is a regular MDI window (via DetachableMdiSubWindow), not a dock, since testers need to move it around freely while operating another dialog under test.
  • All pop-up content spawned from within the panel (step detail, the link viewer) follows the exact same DetachableMdiSubWindow convention as the panel itself, not top-level windows — tried first as detached QDialog=/=QWidget windows with various window-flag workarounds, but window managers kept pinning them "always above" regardless of modality; hosting them as MDI subwindows (matching every existing detail dialog in the app) was both the correct fix and the simpler one.
  • Reflowing a paragraph's wrapped source lines into one logical line is a small, reusable domain utility in ores.orgmode (join_paragraph_lines), not something hand-rolled in the Qt renderer — the raw body_lines already retain blank-line boundaries, so any future consumer needing "real paragraphs, not source-wrapped lines" gets this for free.
  • [[id:...][...]] links inside a rendered doc open in a separate viewer window with its own Back/Forward history, rather than navigating the tab the tester clicked from — following a trail of links shouldn't cost the tester what they were originally looking at.
  • --open-scenario <path> (both on ores.qt directly and via compass client start) exists specifically to make the edit/rebuild/retest loop faster while iterating on this panel — a small tool investment that paid for itself within this one story.

Out of scope

  • Fully automating test execution (this is a manual-test tracking aid, not a test-automation framework).
  • A generic multi-user test-management system; this is scoped to the single-tester, multi-worktree reality described above.
  • A general-purpose org-mode parser/renderer in ores.qt; both the scenario reader/writer and the story/task viewer only need to handle the small, fixed subset of org syntax this project actually uses in its agile docs.
  • The AI-worker-writes-scenario-JSON half of Gemini's loop; scenario docs are hand-authored (by a human or by Claude) for now.

Emacs 29.3 (Org mode 9.6.15)