Story: QA Validation Runner: in-app test tracking for the multi-worktree fleet
Table of Contents
This page documents a story in Sprint 22. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
Goal
Second story of the QA Validation Runner epic (see ores.orgmode: C++ org-mode parsing and org-roam link resolution for the first — this story depends on it and does no org parsing of its own).
The fleet has grown to 10 worktrees, most of them wanting UI-level verification of their in-flight story or task, but there is only one human tester. In practice the tester loses track of what is being tested, on which branch/environment, and what was found — verification notes end up scattered across chat, or not recorded at all.
Add a QA Validation Runner to ores.qt: a dockable panel that turns a
scenario (a checklist of manual steps tied to a story/task) into a
guided, trackable test session, and writes the result back next to the
docs that drove it — not to a JSON file nobody reads again.
Gemini's initial analysis (doc/analysis/gemini_test_ui.org, mockup
doc/analysis/gemini_test_ui_mockup.png) proposes a
test_scenario.json / test_result.json protocol so an AI worker can
hand a checklist to the UI and consume the result back. The panel UX
(docked checklist, PASS/FAIL toggle, freeform notes, "launch target
dialog") is a good starting point, but the storage format changes
direction for this codebase: the scenario is a new org-mode document
type, not JSON.
- A new doc type — provisionally
test_scenario— is scaffolded the same way a task or story is:compass add test_scenario, saved under the driving story's directory, linked (id:link both ways) to the task it verifies. It is filled in with target dialog, title, and the checklist of steps before testing starts — an ordinary org file, reviewable in a PR, greppable, renderable on the site like any other doc. - The Qt panel reads/writes this doc via the
ores.orgmodecomponent from the prerequisite story — it does not implement its own org parsing. What this story adds on top: rewriting the* Resultssection in place on save (a targeted write, not somethingores.orgmode's reader needs to support generically), and the UI itself. - Workflow: instantiate the scenario doc under the story, link it to
the task, fill in the details — then point the Qt panel at that
file. The tester runs through the checklist in the app, ticks steps,
writes notes, sets PASS/FAIL, hits save. The panel rewrites the
scenario doc's
* Resultssection in place (step outcomes, notes, status, environment metadata — branch/commit/worktree identity read from the checkout, not hand-entered) and stamps the compass journal. The doc is now the single source of truth:git diffshows the result, the website renders it, and the next Claude session can just read the file.
A second, closely-related gap: with this much parallel work across the
fleet, the tester (and Claude, picking a session back up) loses the
context of what a given scenario is even testing — the story/task
doc has that context, but nobody wants to leave the running app and go
digging through the site or the org file by hand mid-test. So the same
panel (or a sibling tab) should also render the driving story and its
tasks — title, goal, acceptance, status, tasks table — using the
ores.orgmode document structure to drive a simple, structure-aware
renderer (headings, lists, tables, basic emphasis is enough; not a
faithful org renderer).
The tester also needs a fleet-level view: which worktree is asking for
what test, so they can pick one scenario at a time instead of holding
it all in their head. Compass already renders this info
(compass fleet, compass show <task>) — this story should establish
how the Qt panel surfaces it without duplicating compass's data model.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent sprint | Sprint 22 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | None. |
| Last touched | 2026-07-10 |
Acceptance
[ ]A newtest_scenarioorg doc type is defined (frontmatter, required sections) and scaffoldable viacompass add test_scenario, saved under the driving story's directory andid:-linked to the task it verifies.[ ]ores.qtconsumes the prerequisiteores.orgmodecomponent to parsetest_scenarioand story/task docs — no org parsing reimplemented in Qt. The panel adds only the* Results-section rewrite-on-save and a simple structure-aware renderer for display (headings, lists, tables, basic emphasis).[ ]ores.qtgains a dockable QA Validation Runner panel (debug builds only) that opens atest_scenariodoc, shows its info (title, target dialog, steps), lets the tester tick steps, capture freeform notes, and set a PASS/FAIL outcome.[ ]Environment metadata (branch, commit, worktree identity) is read from the running checkout, not hand-entered by the tester.[ ]Submitting a result rewrites the scenario doc's* Resultssection in place (step outcomes, notes, status, environment metadata) — no standalone JSON artefact — and stamps the compass journal.[ ]The same panel (or a sibling tab) can render the driving story/task docs (goal, acceptance, status, tasks table) so the tester doesn't need to leave the app to recover context.[ ]~The tester can see, from within the panel or a companion list, which worktrees/tasks in the fleet currently have a scenario waiting~— abandoned (see Surface pending scenarios across the fleet): no fleet-coordination source of truth exists yet to build this on; revisit once one does.[ ]Renderedtest_scenariodocs (with their filled-in* Results) show up correctly on the published site, same as any other doc.[ ]Works end-to-end for at least one real story from this sprint as a pilot before being declared done.
Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Define the test_scenario org doc type | DONE | 2026-07-07 | 2026-07-07 | Define and scaffold a new test_scenario org doc type (frontmatter, Scenario Info/Steps/Results sections), following the same doc-type-registration path as story/task, so it's creatable, indexable, and site-renderable like any other agile doc. |
| Wire ores.qt to ores.orgmode and add the Results-rewrite + doc renderer | DONE | 2026-07-08 | 2026-07-08 | Add ores.orgmode as a Qt dependency; implement the targeted * Results section rewrite-on-save for test_scenario docs, and a simple structure-aware renderer (headings/lists/tables/basic emphasis) shared by the scenario panel and the story/task viewer. |
| Build the QA Validation Runner dockable panel | DONE | 2026-07-08 | 2026-07-08 | Qt dockable panel (debug builds only): open a test_scenario doc, show its info and checklist, let the tester tick steps/write notes/set PASS-FAIL, and save (rewriting the doc's Results section and stamping the compass journal). |
| Add the story/task context viewer | DONE | 2026-07-08 | 2026-07-08 | Render the driving story/task docs (goal, acceptance, status, tasks table) in the same panel or a sibling tab, using the shared structure-aware renderer, so the tester doesn't need to leave the app to recover context. |
| Surface pending scenarios across the fleet | ABANDONED | 2026-07-08 | 2026-07-08 | Show which worktrees/tasks in the fleet currently have a scenario waiting, sourced from compass's existing fleet/task data (not duplicated) — a companion list or panel section. Abandoned: no proper fleet coordination exists yet to build this on. |
| Pilot end-to-end on a real sprint story | DONE | 2026-07-10 | Piloted on the Currency Pairs manual chapter screenshot scenario: 5 steps run, Results written back, journal stamped, site rendered correctly. | |
| Add the qa_validation_runner_enabled system setting | DONE | 2026-07-08 | 2026-07-08 | Seed system.qa_validation_runner_enabled as a boolean system setting so the QA Validation Runner's visibility is a runtime, operator-controlled decision, not a compile-time macro. |
| Add screenshot capture to the QA Validation Runner panel | DONE | 2026-07-09 | 2026-07-10 | Add a 'Screenshot' toolbar action next to Open/Save: grab the full primary screen via QScreen::grabWindow(0), save as PNG next to the open scenario doc (same directory as scenarioPath_), named after the scenario slug/step/timestamp. Region-select capture (QRubberBand over a full-screen grab, drag-select, crop, save) as a documented fast-follow, not required for the first cut. |
Decisions
- Scenario and result storage is org-mode docs under the story, not JSON files as Gemini's analysis proposed. Rationale: this project already treats org docs as the source of truth for everything agile — git history, PR review, compass indexing, site rendering all come for free; a JSON sidecar would be a second, disconnected source of truth nobody browses after the fact.
- Org reading/writing is not reimplemented in
ores.qt— it consumes theores.orgmodecomponent from the prerequisite story in this epic, so the parsing investment is made once and shared withores.shelland anything else that needs it. - The panel lives in
ores.qt.admin'sAdminPlugin(contributingSystem > Testingviasetup_menus()), not hardcoded intoMainWindow— it's an administrative/testing tool, the same kind of feature asSystem > Configuration=/=Administration. - Visibility is a runtime system setting
(
system.qa_validation_runner_enabled), not a compile-time#ifdef: a fleet-wide manual-testing tool shouldn't be stripped from release builds, and whether it's available is an operator/deployment decision. Required wideningshared_menus_contextwithmain_window=/=mdi_area=/=client_manager(already constructed beforesetup_menus()runs) since this is the first plugin feature that needs to work before login — narrowing the pre-login gate for that surfaced a separate pre-existing gap (System > Configuration=/=Administrationwere never independently login-gated, only protected by the old blanketSystem-menu disable), fixed alongside it. - Each scenario step is its own org heading (not a bullet checklist)
with its own
*** Resultchild heading, not one aggregate results table — steps need per-item PASS/FAIL/PENDING/notes, and a heading gives each step a natural place to carry both its instructions and its own result. The panel is a regular MDI window (viaDetachableMdiSubWindow), not a dock, since testers need to move it around freely while operating another dialog under test. - All pop-up content spawned from within the panel (step detail, the
link viewer) follows the exact same
DetachableMdiSubWindowconvention as the panel itself, not top-level windows — tried first as detachedQDialog=/=QWidgetwindows with various window-flag workarounds, but window managers kept pinning them "always above" regardless of modality; hosting them as MDI subwindows (matching every existing detail dialog in the app) was both the correct fix and the simpler one. - Reflowing a paragraph's wrapped source lines into one logical line
is a small, reusable domain utility in
ores.orgmode(join_paragraph_lines), not something hand-rolled in the Qt renderer — the rawbody_linesalready retain blank-line boundaries, so any future consumer needing "real paragraphs, not source-wrapped lines" gets this for free. [[id:...][...]]links inside a rendered doc open in a separate viewer window with its own Back/Forward history, rather than navigating the tab the tester clicked from — following a trail of links shouldn't cost the tester what they were originally looking at.--open-scenario <path>(both onores.qtdirectly and viacompass client start) exists specifically to make the edit/rebuild/retest loop faster while iterating on this panel — a small tool investment that paid for itself within this one story.
Out of scope
- Fully automating test execution (this is a manual-test tracking aid, not a test-automation framework).
- A generic multi-user test-management system; this is scoped to the single-tester, multi-worktree reality described above.
- A general-purpose org-mode parser/renderer in
ores.qt; both the scenario reader/writer and the story/task viewer only need to handle the small, fixed subset of org syntax this project actually uses in its agile docs. - The AI-worker-writes-scenario-JSON half of Gemini's loop; scenario docs are hand-authored (by a human or by Claude) for now.